CFEngineとUbuntuの入門チュートリアル

(For the English version click here)

CFEngineとは何ですか。CFEngineは大規模システムの管理に使用できる構成管理システムだ。その元は約束理論ということだが,それによるとすべてのものはシステムのリソースに守られる約束だという。約束とパターンを結び付きによって約束をいつ・どこで実施するか指示するのはエンジンの動作の本質だ。 CFEngineの約束は「*.cf」という拡張子を末尾にある特殊な policy files に守られてる。ネットワークの各ノードはそのファイルを中央ハブあるいはファイル自体(ハブとかネットワークがない場合)から引いて、そのコピーを保存する。当チュートリアルにエンジンのダウンロード、編集とUbutu Linuxにインストールの仕方とノーマル(ノンルート)ユーザーとして使い方の案内したい。システムの実際動作のためにルートユーザーとしてCFEngineを実行しないとできない。なぜならルートユーザーだけが全ての必要な設定変更に許可を持っているからだ。 CFEngineをダウンロードする前にその3つの依存パケージの最新バージョンを持っているか確認する。依存パケージは以下の通りだ:

  • OpenSSL – Open source Secure Sockets Layer for encryption.
  • Tokyo Cabinet (1.4.42あるいはその後のバージョン)
  • PCRE – Perl Compatible Regular Expression library.

依存パケージをインストールに必要な commands は以下の通りだ:

  • TokyoCabinet用:
    sudo apt-get install libtokyocabinet-dev
  • open SSL用:
    sudo apt-get install libssl-dev
  • 最後にPerl Regular Expressions用:
    sudo apt-get install libpcre3-dev

パケージの取得が難しい場合

apt-cache search tokyocabinet

というエントリーを検索したほうが一番いい。 依存パケージからCFEngineソースコードのダウンロードに進もう。ダウンロードのディレクトリーに進む ( /Path/To はダウンロード先のディレクトリーで、x.yはダウンロードしたバージョンの番号だ。次、順番に以下のcommandsをインプットする:

cd /Path/To/cfenfine-3.x.y
./configure
make
sudo make install

上記の ./configure はシステムがエンジンを築けるか確認し、makeはエンジンを築いて適切なディレクトリーにインストールする。 CFEngineはworking directoryという概念を使用して動作する。Linuxでルートユーザー用の既定working directoryは /var/cfengine だ。その他のユーザーの場合 ~/.cfagentで探す。CFEngineをUbuntuでノンルートユーザーとしてテストだけするつもりなのでまず、次のステップを実施する必要だ:

  • 要求したディレクトリーがない場合 ~/.cfagentにそれを作る。(僕にはあった)
    mkdir -p ~/.cfagent/inputs
    mkdir -p ~/.cfagent/bin
  • ルートCFEngineのbinの内容をコピーして、ユーザーのworking directory binに貼り付ける
    cp /var/cfengine/bin/cf-* ~/.cfagent/bin

今は例のpolicy fileを見てみよう。CFEngineのHello Worldを。このpolicy fileを ~/.cfagent/inputs に入れるのに次のcommandsを使う:

touch ~/.cfagent/inputs/test.cf
chmod 600 ~/.cfagent/inputs/test.cf
echo -e "body common control
{
bundlesequence => { \"test\" };
}
bundle agent test
{
reports:
  cfengine_3::
      \"Hello world\";
}" >> ~/.cfagent/inputs/test.cf

上記のcommandsでテストpolicy fileを作成し、そのパーミッションを変更し、公式ガイドから複写したHello world policy を貼り付ける。パーミッションを変える理由は policy files の要求だ。CFEngineを実行するユーザーだけの所属でグループとかその他にパーミッションがあればsecurity exceptionが発生してcf-promises から警報が出る。 それから policy file の構文的に正しいか確認のために cf-promises を使用する.

cf-promises -f ~/.cfagent/inputs/test.cf

正しければ画面のアウトプットが何もないけど逆の場合エラーのメッセージが表示される。 最後に policy file を実行するように cf-agent を使う。

cf-agent -f ~/.cfagent/inputs/test.cf

出力は下記のようになる:

R: Hello World

手動で cf-agent を実行したのに普通は cf engine daemon, cf-execd に一定間隔(5分は既定)で起動される。そして policy file の正しさのテストのため cf-agent が自動的に cf-promises を呼び出す。 Policy fileに進んで構文を見てみよう。

body common control 
{
bundlesequence => { "test" };
}

上記はC/C++/Java のmain() functionに似ている。本文に下記のようなbundle agent testを呼び出す。

bundle agent test  
{
reports:
  cfengine_3::
   "Hello world!";
}

約束のアトリビュートの収集だ。約束が守られたなら”Hello world!”を出力する。上記の約束の意味はシステムがCFEngineのバージョン3以上を実行すると出力が”Hello World”になるということ。 この入門ポリシーができて理解したら通常に要求される動作のポリシー作成に使えるPolicy Wizardに進める。ちょっとそれで遊んでみてCFEngineの約束の構文で実行できる動作を見るのは面白い練習になる。 今、CFEngineエージェントを詳細に見よう。でもノーマル(ノンルート)ユーザーとしてCFEngineをテストしてるのでその前pre-configurationをしたほうがいい 。

cp -R /var/cfengine/share/CoreBase/* ~/.cfagent/inputs

上記がスタンダード CFEngine policy files を現在ユーザーの動作ディレクトリーのinputs policy filesに複写する。なぜなら、cf-agent が WORKINGDIR/inputs に所属している promises.cf という既定のpolicy fileをチェックする.。 Policy file を開いたら、Hello Worldファイルと同種な構文だとみえる。

###############################################################################
#
#   promises.cf - Basic Policy for Community
#
###############################################################################
 
body common control
 
{
 
 bundlesequence => {
 
                 # Common bundles first for best practice 
                    "def",
 
                 # Design Center
                    "cfsketch_run",
 
                 # Agent buddles from here
                    "main"
                   };
 
 inputs => {
 
         # Global common bundles
            "def.cf",
 
         # Control body for all agents
            "controls/cf_agent.cf",
            "controls/cf_execd.cf",
            "controls/cf_monitord.cf",
            "controls/cf_report.cf",
            "controls/cf_runagent.cf",
            "controls/cf_serverd.cf",
 
         # COPBL/Custom libraries
            "libraries/cfengine_stdlib.cf",
 
         # Design Center
             # MARKER FOR CF-SKETCH INPUT INSERTION
             "cf-sketch-runfile.cf",
 
         # User services from here
            "services/init_msg.cf",
 
           };
 
 version => "Community Promises.cf 3.4.0";
 
}
 
###############################################################################
 
bundle agent main
{
 
 methods:
 
  any::
   "INIT MSG" usebundle => init_msg,
                comment => "Just a pre-defined policy bundled with the package",
                 handle => "main_methods_any_init_msg";
}
 
###############################################################################

最初から: body common control は policy file の本体を規定する。#で始まるものはコメントであり Hello World の例と同様に bundle sequence を呼び出す。今シーケンスの bundle 数は1つ以上だ。Bundleは同じ policy file (主ファイルを例にして)あるいはinputs=>に所属してるその他のpolicy files二規定される。Bundlesがシーケンスに呼び出されているんだが、上記の “def” は最初、”main” は最終に呼び出されている。 Inputs に所定の policy files があって、その内容はメインポリシーに含んでいる機能とか定義だ。自分のを追加してみないか。

touch ~/.cfagent/inputs/fileTest.cf
chmod 600 ~/.cfagent/inputs/fileTest.cf

以上の通り、新しいpolicy fileを作成して以下のコードをコピーしてファイルに貼り付ける。

bundle agent FileTester
{
files: #file related promise
 
"/tmp/testityfile" #promiser file
 
    create => "true",   # If the file does not exist, create it
    perms => m("600"), # Make sure that the permissions are only read/write by the owner
    edit_line => replace_or_add("","I should exist in the file"); #Search the file for the given string, if it is not found replace an empty string with it
 
reports: #also adding a report to see something in stdout
    cfengine_3::  
        "File \"testityfile\" has been edited";
}

これは単に別の bundle agent だ。まず、ファイル関連の約束だと規定する。次、promiser (約束先) file名を記入する。最後に約束の3つの属性を規定する。つまり:

  • ファイルがないなら、ファイルを作る
  • ファイルのパーミッションは600ではなければパーミッションを変える
  • ファイルの中に “I should exist in the file” がなければ空いてる””ストリングにそれを書く

今、既定 promises.cf ファイルに戻って bundle sequence に自分のbundleを作ってみよう

 bundlesequence => {
 
                 # Common bundles first for best practice 
                    "def",
 
                 # Design Center
                    "cfsketch_run",
 
                 # Agent buddles from here
                    "main",
 
                 # This is our files testing functionality
                    "FileTester"
 
                   };

できたbundleをinputsに入れてその行方を明確にする。

 inputs => {
 
         # Global common bundles
            "def.cf",
 
         # Control body for all agents
            "controls/cf_agent.cf",
            "controls/cf_execd.cf",
            "controls/cf_monitord.cf",
            "controls/cf_report.cf",
            "controls/cf_runagent.cf",
            "controls/cf_serverd.cf",
 
         # COPBL/Custom libraries
            "libraries/cfengine_stdlib.cf",
 
         # Design Center
             # MARKER FOR CF-SKETCH INPUT INSERTION
             "cf-sketch-runfile.cf",
 
         # User services from here
            "services/init_msg.cf",
         # Specify the policy file for our file testing
            "fileTest.cf"
 
           };

今、cf-agent を起動してみると tmp ディレクトリに testityfile ができ その役割は約束を守ることだ。ファイルの変更にかかわらずに cf-agent が実行する時、policy file の約束が守られてる。 Policy file に replace_or_add() があったが、標準ポリシーのライブラリー機能の例だ。すべてはinputsに含まれる cfengine_stdlib.cf に所属するんだ。つまらない例なんだけど標準ライブラリーに提供される機能は精鋭でシステムの管理に有能なツールになる。例えば:

  • edit line => set_user_field(user,field,val): “field”というフィールド番号の値を/etc/passwdのような:-field formatted fileに設定する
  • edit line => append_user_field(group,field,allusers): /etc/groupのようなファイルにユーザーを追加する

標準ライブラリの policy file で他の機能とその役割も見える。 そいうことで。上記はとても基準的なCFEngineの案内でそのツールとしての能力を表面的な説明しかできなかった。次はルートユーザーに切り替えて動いてみること。これはエンジンの対象の実行方法だから。そして外部サーバー(ハブ)からポリシーを引くのはエンジンのりそうな使用方法だ。詳細をofficial CFEngine 3 tutorialに参考ください。 時間があれば、次のCFEngineチュートリアルも書いてみたい。コメント・フィードバックがあればそれとも僕の間違いを気づければぜひ教えてください(コメントあるいは直接、メールで)。 Stay sharp!

An introductory tutorial to CFEngine and Ubuntu

An introductory tutorial to using CFEngine with Ubuntu Linux. CFEngine is a configuration management system utilized in large system administration. It uses promise theory according to which everything can be thought of as a promise to be kept by different resources in the system. Combining promises with patterns to describe where and when promises should apply is the essence of the engine’s operation. We will see how compile and install the engine in Ubuntu Linux and how to run a few simple examples.

This is an introductory tutorial to CFEngine. There is also a Japanese version of the article which can be found here. 日本版はこちらへ.

What is CFEngine? CFEngine is a configuration management system that can be utilized in large system administration. It is based on promise theory, according to which everything can be thought of as a promise to be kept by different resources in the system. Combining promises with patterns to describe where and when promises should apply is the essence of the engine’s operation.

CFEngine’s promises are kept in special files called policy files and ending in *.cf. Each node of the network keeps a copy of these files which it pulls from a central hub or from itself in the absence of a network (or a hub). In this tutorial we will simply see how to download, compile and install the engine in an Ubuntu Linux system and how to use it as a normal (non-root) user. The reader has to note that for a real operation of the system you would have to run CFEngine as the root user because this is the only way that you could grant it the permission to perform any configuration change that may be needed.

Before obtaining CFengine we have to make sure that we have got the latest versions of the 3 dependencies of the engine. They are:

  • OpenSSL – Open source Secure Sockets Layer for encryption.
  • Tokyo Cabinet (version 1.4.42 or later)
  • PCRE – Perl Compatible Regular Expression library.

Below you can see the commands needed to install the dependent packages at the time of the writing of this article:

  • For TokyoCabinet:
    sudo apt-get install libtokyocabinet-dev
  • For open SSL:
    sudo apt-get install libssl-dev
  • And finally for the Perl Regular Expressions:
    sudo apt-get install libpcre3-dev

If you have any problems obtaining any of these packages then simply perform a search using

apt-cache search tokyocabinet

to find the exact package name available and then download it.

By now all our dependencies have been taken care of so you can proceed to download the source code for CFEngine. Navigate to the downloaded directory (where /Path/To is the directory where you downloaded the source code and x.y the actual version number you downloaded. Then type the following commands one by one.

cd /Path/To/cfenfine-3.x.y
./configure
make
sudo make install

With the above ./configure makes sure our system is ready to build the engine and the make commands simply build it and install everything into the appropriate directories.

CFEngine operates using the notion of a working directory. The default working directory for the root user in Linux is /var/cfengine. For any other user the working directory can be found in ~/.cfagent. In our case since we simply want to test CFEngine in Ubuntu and not as the root user we need to perform some steps first.

  • Make the required directories under ~/.cfagent if they don’t exist. (In my case they already did)
    mkdir -p ~/.cfagent/inputs
    mkdir -p ~/.cfagent/bin
  • Copy everything from the root’s CFEngine bin to the user’s working directory bin
    cp /var/cfengine/bin/cf-* ~/.cfagent/bin

Now let us take a look at a sample policy file. The Hello World of CFengine. We will place this sample policy file inside ~/.cfagent/inputs with the following commands.

touch ~/.cfagent/inputs/test.cf
chmod 600 ~/.cfagent/inputs/test.cf
echo -e "body common control
{
bundlesequence => { \"test\" };
}
bundle agent test
{
reports:
  cfengine_3::
      \"Hello world\";
}" >> ~/.cfagent/inputs/test.cf

With the above we create a test policy file, change its permissions and finally copy the Hello world policy from the official guide into it. The permission change is done due to a requirement that policy files have. They should only belong to the user who runs CFEngine. No Group or Others permission should exist. If they do, this would result in a security exception and cf-promises would warn us about it

Now we have to verify that the policy file is syntactically correct. To do this we use cf-promises.

cf-promises -f ~/.cfagent/inputs/test.cf

This should output nothing in the screen if all is okay with the policy file. Otherwise appropriate errors would be printed.

Finally to execute the policy file manually we use cf-agent

cf-agent -f ~/.cfagent/inputs/test.cf

This should output:

R: Hello World

Even though we manually executed cf-agent normally it would be invoked by the cf engine daemon, cf-execd, in given internals (the default is every 5 minutes). Additionally cf-agent automatically invokes cf-promises in order to test that a policy file is correctly written.

But let’s go in the above policy file and have a look at the syntax.

body common control 
{
bundlesequence => { "test" };
}

The above is akin to the main() function in a C/C++/Java program. Inside the body we are simply calling the bundle agent test which is defined below.

bundle agent test  
{
reports:
  cfengine_3::
   "Hello world!";
}

It is a collection of promise attributes. It will output “Hello world!” if all the promises are kept. The promise that we wrote above simply means that if the system is running CFEngine version 3 and above you should output “Hello World”.

With this introductory policy done and understood you can also proceed to the the Policy Wizard which can help you with creating policies for commonly required operations. It is a nice exercise playing around with it and seeing how different operations can be achieved with the CFEngine promises syntax.

Now let’s look at the CFEngine agent in more detail. But before we do that we need to do a pre-confgiuration step since we are still testing CFEngine as a normal user and not as the root user.

cp -R /var/cfengine/share/CoreBase/* ~/.cfagent/inputs

The above will copy the standard CFEngine policy files to the inputs policy files of the current user’s working directory. Why do this? Well, cf-agent checks a default policy file and that file is promises.cf located at WORKINGDIR/inputs.

Open the policy file and you will notice the same kind of syntax as we had in the hello World file.

###############################################################################
#
#   promises.cf - Basic Policy for Community
#
###############################################################################
 
body common control
 
{
 
 bundlesequence => {
 
                 # Common bundles first for best practice 
                    "def",
 
                 # Design Center
                    "cfsketch_run",
 
                 # Agent buddles from here
                    "main"
                   };
 
 inputs => {
 
         # Global common bundles
            "def.cf",
 
         # Control body for all agents
            "controls/cf_agent.cf",
            "controls/cf_execd.cf",
            "controls/cf_monitord.cf",
            "controls/cf_report.cf",
            "controls/cf_runagent.cf",
            "controls/cf_serverd.cf",
 
         # COPBL/Custom libraries
            "libraries/cfengine_stdlib.cf",
 
         # Design Center
             # MARKER FOR CF-SKETCH INPUT INSERTION
             "cf-sketch-runfile.cf",
 
         # User services from here
            "services/init_msg.cf",
 
           };
 
 version => "Community Promises.cf 3.4.0";
 
}
 
###############################################################################
 
bundle agent main
{
 
 methods:
 
  any::
   "INIT MSG" usebundle => init_msg,
                comment => "Just a pre-defined policy bundled with the package",
                 handle => "main_methods_any_init_msg";
}
 
###############################################################################

In the beginning body common control defines the body of our policy file. Anything starting with # is a comment and just like in the Hello World example we have a bundle sequence that we are calling. Only now there are more than one bundles in the sequence. These bundles can either be defined in the same policy file (main is for example) or at any of the other policy files that are defined at inputs =>. The bundles are called in sequence, so above “def” would be called first and “main” would be the last.

In inputs we can see many of the predefined policy files, containing functions and definitions that the main policy includes. Why don’t we check how we can add one of our own?

touch ~/.cfagent/inputs/fileTest.cf
chmod 600 ~/.cfagent/inputs/fileTest.cf

Create a new policy file as shown above and copy the code below inside it

bundle agent FileTester
{
files: #file related promise
 
"/tmp/testityfile" #promiser file
 
    create => "true",   # If the file does not exist, create it
    perms => m("600"), # Make sure that the permissions are only read/write by the owner
    edit_line => replace_or_add("","I should exist in the file"); #Search the file for the given string, if it is not found replace an empty string with it
 
reports: #also adding a report to see something in stdout
    cfengine_3::  
        "File \"testityfile\" has been edited";
}

What will this do? This is just another bundle agent. At the beginning we specify that it is a files related promise. Then we provide the name of the promiser file. Afterwards we specify the 3 attributes of the promise. In simple English:

  • If the file does not exist, create it
  • If the file’s permissions are not 600 then change them
  • If “I should exist in the file” is not inside the file then replace the empty string “” with it

Now go back to the default promises.cf file and add our very own bundle into the bundle sequence.

 bundlesequence => {
 
                 # Common bundles first for best practice 
                    "def",
 
                 # Design Center
                    "cfsketch_run",
 
                 # Agent buddles from here
                    "main",
 
                 # This is our files testing functionality
                    "FileTester"
 
                   };

Also specify where the engine can find this bundle by adding it to the inputs

 inputs => {
 
         # Global common bundles
            "def.cf",
 
         # Control body for all agents
            "controls/cf_agent.cf",
            "controls/cf_execd.cf",
            "controls/cf_monitord.cf",
            "controls/cf_report.cf",
            "controls/cf_runagent.cf",
            "controls/cf_serverd.cf",
 
         # COPBL/Custom libraries
            "libraries/cfengine_stdlib.cf",
 
         # Design Center
             # MARKER FOR CF-SKETCH INPUT INSERTION
             "cf-sketch-runfile.cf",
 
         # User services from here
            "services/init_msg.cf",
         # Specify the policy file for our file testing
            "fileTest.cf"
 
           };

Now if you invoke cf-agent you will get a file called testityfile under the /tmp directory which will keep all the promises of our policy file. No matter what change you (or anyone else) makes to the file, whenever cf-agent runs the promises of our policy file will be kept.

In the policy file we saw replace_or_add() which is one example a standard policy library function. They are all located in cfengine_stdlib.cf which as you may notice is included in the inputs. Our example is quite silly but the functions offered by the standard library are quite powerful and tailored to help with system administration needs. For example:

  • edit line => set_user_field(user,field,val): Sets the value of field number “field” in a :-field formatted file like /etc/passwd
  • edit line => append_user_field(group,field,allusers): Adds users to a file like /etc/group

You can check the standard library policy file itself to see examples of more functions and what they can achieve.

Well that’s all. This is but a very soft introduction to CFEngine and we have barely scratched the surface of what this tool is capable of doing. Next up would be to actually switch to doing everything as the root user, since this is the only way the engine should be ran. Also attempting to pull the policies from an external server (a hub) is a must for the ideal engine usage. For that and for much more visit the official CFEngine 3 tutorial

If I find more free time, more advanced CFEngine tutorials may follow. If you have any comments/feedback or spotted mistakes in this article please don’t hesitate to leave a comment or contact me directly. Stay sharp!