fluentd + elasticsearch + Kibana3 によるログ収集・可視化

温度計などのセンサデータを、MA-E3xx により収集し、fluentd により elasticsearch に集約し、
Kibana3 により可視化を行ってみます。
今流行りの BuzzWord, “IoT” っていう感じでしょうか。

構成イメージはこのようになります。

(画像ファイルは公式サイトより流用)

シリアルポートのデータを、TCP 変換してサーバに接続するというのは簡単だとは思いますが、
3G 回線が切れたときなど、通信ができないときのデータ欠損が避けらず、オススメできません。
シリアルポート通信はリモート側で終端し、この仕組みでデータを送信するのが今時の仕組みではないでしょうか。

参考

書籍

最高の参考書が発売されました。

Web

Fluentdとログ収集のパターン (Go ahead! - Memoization for Everything)
柔軟なログ収集を可能にする「fluentd」入門 (さくらのナレッジ)
研究開発用データセンターの空調制御効率化を目指したデータ収集 (IIJ)
Fluentdによる研究開発用DCのデータ収集 (slideshare) (上記 IIJ のプレゼン資料, Vaisala 使ってます)
Fluentd + ElasticSearch + Kibana3で簡単に様々なログを可視化・解析する (404 Engineer Logs)
Kibana3というのもありまして (@johtaniの日記 2nd)
Elasticsearch, Fluentd, and Kibana: Open Source Log Search and Visualization (DigitalOcean)
ログ収集基盤 fluentdのご紹介(PDF) (SRA OSS)
FluentdがKubernetesの標準ログ収集ツールとして採用 (Publickey)

準備

MA-E3xx 側には、fluentd をインストールする必要があります。残念ながら、Ubuntu14.04 用の公式パッケージは用意されておりません。
パッケージでインストールしておかないと後々管理ができなくなりますので、 Mario FetkaさんのPPA https://launchpad.net/~mario-fetka/+archive/fluentd
からパッケージを作成し、インストールすることにします。

パッケージ作成にあたっては、下記環境が前提となります。

開発環境一式 (build-essential) をインストール済みであること。¹⁾
パッケージ作成用のパッケージ(devscripts)をインストール済みであること。²⁾
Ruby 開発環境 (ruby-dev) をインストール済みであること。

※ fluentd を組み込んだファームウェアを、“MA-E3xx for IoT” に置いてあります。

Ruby 開発環境のセットアップ

Ruby2.0 を使用したい³⁾ので、ruby2.0, ruby2.0-dev をインストールします。

user1@plum:~$ sudo apt-get install ruby2.0 ruby2.0-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libjs-jquery libruby1.9.1 libruby2.0 libyaml-0-2 ruby ruby1.9.1
  rubygems-integration
Suggested packages:
  javascript-common ri ruby-dev ruby1.9.1-examples ri1.9.1 graphviz
  ruby1.9.1-dev ruby-switch bundler
The following NEW packages will be installed:
  libjs-jquery libruby1.9.1 libruby2.0 libyaml-0-2 ruby ruby1.9.1 ruby2.0
  ruby2.0-dev rubygems-integration
0 upgraded, 9 newly installed, 0 to remove and 2 not upgraded.
Need to get 6089 kB of archives.
After this operation, 25.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libyaml-0-2 armhf 0.1.4-3ubuntu3 [40.4 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libjs-jquery all 1.7.2+dfsg-2ubuntu1 [78.8 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby all 1:1.9.3.4 [5334 B]
Get:4 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby1.9.1 armhf 1.9.3.484-2ubuntu1 [35.5 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libruby1.9.1 armhf 1.9.3.484-2ubuntu1 [2458 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libruby2.0 armhf 2.0.0.484-1ubuntu2 [2599 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby2.0 armhf 2.0.0.484-1ubuntu2 [66.3 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports/ trusty/main rubygems-integration all 1.5 [5340 B]
Get:9 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby2.0-dev armhf 2.0.0.484-1ubuntu2 [800 kB]
Fetched 6089 kB in 10s (590 kB/s)                                              
Selecting previously unselected package libyaml-0-2:armhf.
(Reading database ... 21471 files and directories currently installed.)
Preparing to unpack .../libyaml-0-2_0.1.4-3ubuntu3_armhf.deb ...
Unpacking libyaml-0-2:armhf (0.1.4-3ubuntu3) ...
Selecting previously unselected package libjs-jquery.
Preparing to unpack .../libjs-jquery_1.7.2+dfsg-2ubuntu1_all.deb ...
Unpacking libjs-jquery (1.7.2+dfsg-2ubuntu1) ...
Selecting previously unselected package ruby.
Preparing to unpack .../ruby_1%3a1.9.3.4_all.deb ...
Unpacking ruby (1:1.9.3.4) ...
Selecting previously unselected package ruby1.9.1.
Preparing to unpack .../ruby1.9.1_1.9.3.484-2ubuntu1_armhf.deb ...
Unpacking ruby1.9.1 (1.9.3.484-2ubuntu1) ...
Selecting previously unselected package libruby1.9.1.
Preparing to unpack .../libruby1.9.1_1.9.3.484-2ubuntu1_armhf.deb ...
Unpacking libruby1.9.1 (1.9.3.484-2ubuntu1) ...
Selecting previously unselected package libruby2.0:armhf.
Preparing to unpack .../libruby2.0_2.0.0.484-1ubuntu2_armhf.deb ...
Unpacking libruby2.0:armhf (2.0.0.484-1ubuntu2) ...
Selecting previously unselected package ruby2.0.
Preparing to unpack .../ruby2.0_2.0.0.484-1ubuntu2_armhf.deb ...
Unpacking ruby2.0 (2.0.0.484-1ubuntu2) ...
Selecting previously unselected package rubygems-integration.
Preparing to unpack .../rubygems-integration_1.5_all.deb ...
Unpacking rubygems-integration (1.5) ...
Selecting previously unselected package ruby2.0-dev:armhf.
Preparing to unpack .../ruby2.0-dev_2.0.0.484-1ubuntu2_armhf.deb ...
Unpacking ruby2.0-dev:armhf (2.0.0.484-1ubuntu2) ...
Setting up libyaml-0-2:armhf (0.1.4-3ubuntu3) ...
Setting up libjs-jquery (1.7.2+dfsg-2ubuntu1) ...
Setting up ruby2.0-dev:armhf (2.0.0.484-1ubuntu2) ...
Setting up ruby (1:1.9.3.4) ...
Setting up ruby1.9.1 (1.9.3.484-2ubuntu1) ...
Setting up libruby1.9.1 (1.9.3.484-2ubuntu1) ...
Setting up rubygems-integration (1.5) ...
Setting up ruby2.0 (2.0.0.484-1ubuntu2) ...
Setting up libruby2.0:armhf (2.0.0.484-1ubuntu2) ...
Processing triggers for libc-bin (2.19-0ubuntu6) ...
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

user1@plum:~$

上のとおり、ruby1.9.1 も一緒にインストールされてしまいます。

user1@plum:~$ ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [arm-linux-eabihf]
user1@plum:~$

単に “ruby” として実行すると、Ruby1.9 系がデフォルトとなっているため、Ruby2.0 系をデフォルトにする設定をします。
参考: ´・ω・)＜ときどき書くよ - Ubuntu 14.04 LTS に Fluentd をインストールする

user1@plum:~$ sudo update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.9.1 0
update-alternatives: using /usr/bin/ruby1.9.1 to provide /usr/bin/ruby (ruby) in auto mode
user1@plum:~$ sudo update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby2.0 1
update-alternatives: using /usr/bin/ruby2.0 to provide /usr/bin/ruby (ruby) in auto mode
user1@plum:~$ ruby -v
ruby 2.0.0p384 (2014-01-12) [arm-linux-gnueabihf]
user1@plum:~$

Gem も同様に 2.0 系をデフォルトに変更します。

user1@plum:~$ sudo update-alternatives --install /usr/bin/gem gem /usr/bin/gem1.9.1 0
update-alternatives: using /usr/bin/gem1.9.1 to provide /usr/bin/gem (gem) in auto mode
user1@plum:~$ sudo update-alternatives --install /usr/bin/gem gem /usr/bin/gem2.0 1
update-alternatives: using /usr/bin/gem2.0 to provide /usr/bin/gem (gem) in auto mode
user1@plum:~$ gem -v
2.0.14
user1@plum:~$

パッケージの作成

PPA の追加

PPA を追加します。

user1@plum:~$ sudo add-apt-repository ppa:mario-fetka/fluentd
[sudo] password for user1: 
 Fluentd is an open source data collector designed for processing data streams. 
 More info: https://launchpad.net/~mario-fetka/+archive/ubuntu/fluentd
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpavrdryx3/secring.gpg' created
gpg: keyring `/tmp/tmpavrdryx3/pubring.gpg' created
gpg: requesting key F1AA2FAE from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpavrdryx3/trustdb.gpg: trustdb created
gpg: key F1AA2FAE: public key "Launchpad PPA for Mario Fetka" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
user1@plum:~$

バイナリパッケージが準備されていないものがありますので、PPA のソースパッケージの項を有効にします。

user1@plum:~$ sudo nano -w /etc/apt/sources.list.d/mario-fetka-fluentd-trusty.list

deb http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu trusty main
# deb-src http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu trusty main  <--

“deb-src” の前のコメントを外して保存します。

追加された PPA のパッケージ情報を取得するため、apt-get update によりパッケージ情報を更新します。

user1@plum:~$ sudo apt-get update
〜〜〜 略 〜〜〜
Get:1 http://ppa.launchpad.net trusty/main Sources [12.6 kB]          
〜〜〜 略 〜〜〜
Fetched 12.6 kB in 7s (1590 B/s)                                               
Reading package lists... Done
user1@plum:~$

パッケージの確認

“fluentd” 関連のパッケージを検索してみます。

user1@plum:~$ sudo apt-cache search fluentd
ruby-fluentd - Fluentd event collector
ruby-fluent-mixin-config-placeholders - Configuration syntax extension mixin for fluentd plugin
ruby-fluent-mixin-plaintextformatter - Text formatter mixin module to create fluentd plugin
ruby-fluent-mixin-rewrite-tag-name - Fluentd mixin plugin to provides placeholder function for rewriting tag for your any plugins as like fluent-plugin-rewrite-tag-filter. It will let you get easy to implement tag placeholder for your own plugins.
ruby-fluent-plugin-config-expander - Fluentd plugin to serve some DSL directives in configuration
ruby-fluent-plugin-event-tail - fluentd input plugin for reading [tag, time, record] messages from a file
ruby-fluent-plugin-file-sprintf - sprintf output file plugin for Fluentd.
ruby-fluent-plugin-flatten - Fluentd plugin to extract values for nested key paths and re-emit them as flat tag/record pairs.
ruby-fluent-plugin-geoip - Fluentd Output plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.
ruby-fluent-plugin-glusterfs - Fluentd plugin for GlusterFS
ruby-fluent-plugin-grep - fluentd plugin to grep messages
ruby-fluent-plugin-in-udp-event - Event driven udp input plugin for fluentd
ruby-fluent-plugin-record-reformer - Fluentd plugin to add or replace fields of a event record
ruby-fluent-plugin-reemit - Fluentd plugin to re-emit messages avoiding infinity match loop
ruby-fluent-plugin-rewrite - Fluentd plugin to rewrite tags/values along with pattern matching and re-emit them.
ruby-fluent-plugin-rewrite-tag-filter - Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite. Re-emmit a record with rewrited tag when a value matches/unmatches with the regular expression. Also you can change a tag from apache log by domain, status-code(ex. 500 error), user-agent, request-uri, regex-backreference and so on with regular expression.
user1@plum:~$

パッケージのビルド

Fluentd 本体は “ruby-fluentd” ですが、下記パッケージに依存しています。

ruby-iobuffer
ruby-cool.io

両方共 ARM 用バイナリパッケージが準備されていませんので、ソースパッケージをダウンロードしてビルドを行います。

ruby-iobuffer のビルド

user1@plum:~$ sudo apt-get source ruby-iobuffer
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Need to get 9663 B of source archives.
Get:1 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-iobuffer 1.1.2-2ppa1~trusty1 (tar) [6394 B]
Get:2 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-iobuffer 1.1.2-2ppa1~trusty1 (diff) [1949 B]
Get:3 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-iobuffer 1.1.2-2ppa1~trusty1 (dsc) [1320 B]
Fetched 9663 B in 1s (8186 B/s)    
gpgv: Signature made Fri Apr 18 19:24:59 2014 JST using DSA key ID 0F0C7602
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./ruby-iobuffer_1.1.2-2ppa1~trusty1.dsc
dpkg-source: info: extracting ruby-iobuffer in ruby-iobuffer-1.1.2
dpkg-source: info: unpacking ruby-iobuffer_1.1.2.orig.tar.gz
dpkg-source: info: unpacking ruby-iobuffer_1.1.2-2ppa1~trusty1.debian.tar.gz
user1@plum:~$

user1@plum:~$ cd ruby-iobuffer-1.1.2 
user1@plum:~/ruby-iobuffer-1.1.2$ sudo debuild -us -uc -b
 dpkg-buildpackage -rfakeroot -D -us -uc -b
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: source package ruby-iobuffer
dpkg-buildpackage: source version 1.1.2-2ppa1~trusty1
dpkg-buildpackage: source distribution trusty
dpkg-buildpackage: source changed by Mario Fetka <mario@debian7.disconnected-by-peer.at>
 dpkg-source --before-build ruby-iobuffer-1.1.2
dpkg-buildpackage: host architecture armhf
dpkg-source: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-checkbuilddeps: Unmet build dependencies: debhelper (>= 7.0.50~) gem2deb (>= 0.3.0~) rake rake-compiler ruby-rspec
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1364:
dpkg-buildpackage -rfakeroot -D -us -uc -b failed
user1@plum:~/ruby-iobuffer-1.1.2$

上の通り、いくつかのパッケージがビルドには必要となります。提示されたパッケージをインストールします。
“fakeroot” も必要となりますのでインストールしておきます。

user1@plum:~/ruby-iobuffer-1.1.2$ sudo apt-get install debhelper gem2deb rake rake-compiler ruby-rspec fakeroot
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  bsdmainutils dh-apparmor gettext gettext-base groff-base intltool-debian
  libasprintf0c2 libcroco3 libfakeroot libunistring0 man-db po-debconf
  ruby-diff-lcs ruby-rspec-core ruby-rspec-expectations ruby-rspec-mocks
  ruby1.9.1-dev
Suggested packages:
  wamerican wordlist whois vacation dh-make apparmor-easyprof gettext-doc
  groff www-browser libmail-box-perl
Recommended packages:
  libasprintf-dev libgettextpo-dev libmail-sendmail-perl
The following NEW packages will be installed:
  bsdmainutils debhelper dh-apparmor fakeroot gem2deb gettext gettext-base
  groff-base intltool-debian libasprintf0c2 libcroco3 libfakeroot
  libunistring0 man-db po-debconf rake rake-compiler ruby-diff-lcs ruby-rspec
  ruby-rspec-core ruby-rspec-expectations ruby-rspec-mocks ruby1.9.1-dev
0 upgraded, 23 newly installed, 0 to remove and 2 not upgraded.
Need to get 5179 kB of archives.
After this operation, 16.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main libasprintf0c2 armhf 0.18.3.1-1ubuntu3 [6488 B]
Get:2 http://ports.ubuntu.com/ubuntu-ports/ trusty/main groff-base armhf 1.22.2-5 [934 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports/ trusty/main bsdmainutils armhf 9.0.5ubuntu1 [195 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports/ trusty/main man-db armhf 2.6.7.1-1 [832 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libcroco3 armhf 0.6.8-2ubuntu1 [69.1 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libunistring0 armhf 0.9.3-5ubuntu3 [251 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main gettext-base armhf 0.18.3.1-1ubuntu3 [45.8 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main gettext armhf 0.18.3.1-1ubuntu3 [779 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports/ trusty/main intltool-debian all 0.35.0+20060710.1 [31.6 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports/ trusty/main po-debconf all 1.0.16+nmu2ubuntu1 [210 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports/ trusty/main dh-apparmor all 2.8.95~2430-0ubuntu5 [11.3 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports/ trusty/main debhelper all 9.20131227ubuntu1 [604 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports/ trusty/main libfakeroot armhf 1.20-3ubuntu2 [21.1 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports/ trusty/main fakeroot armhf 1.20-3ubuntu2 [55.2 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby1.9.1-dev armhf 1.9.3.484-2ubuntu1 [784 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports/ trusty/main gem2deb all 0.6.1 [41.8 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports/ trusty/main rake all 10.0.4-1 [49.0 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe rake-compiler all 0.9.2-1 [17.4 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-diff-lcs all 1.2.4-1 [26.9 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-rspec-expectations all 2.14.2-1 [57.9 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-rspec-mocks all 2.14.3-1 [52.7 kB]
Get:22 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-rspec-core all 2.14.7-2 [97.0 kB]
Get:23 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-rspec all 2.14.1-1 [4996 B]
Fetched 5179 kB in 14s (353 kB/s)                                              
Preconfiguring packages ...
Selecting previously unselected package libasprintf0c2:armhf.
(Reading database ... 23420 files and directories currently installed.)
Preparing to unpack .../libasprintf0c2_0.18.3.1-1ubuntu3_armhf.deb ...
Unpacking libasprintf0c2:armhf (0.18.3.1-1ubuntu3) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../groff-base_1.22.2-5_armhf.deb ...
Unpacking groff-base (1.22.2-5) ...
Selecting previously unselected package bsdmainutils.
Preparing to unpack .../bsdmainutils_9.0.5ubuntu1_armhf.deb ...
Unpacking bsdmainutils (9.0.5ubuntu1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../man-db_2.6.7.1-1_armhf.deb ...
Unpacking man-db (2.6.7.1-1) ...
Selecting previously unselected package libcroco3:armhf.
Preparing to unpack .../libcroco3_0.6.8-2ubuntu1_armhf.deb ...
Unpacking libcroco3:armhf (0.6.8-2ubuntu1) ...
Selecting previously unselected package libunistring0:armhf.
Preparing to unpack .../libunistring0_0.9.3-5ubuntu3_armhf.deb ...
Unpacking libunistring0:armhf (0.9.3-5ubuntu3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../gettext-base_0.18.3.1-1ubuntu3_armhf.deb ...
Unpacking gettext-base (0.18.3.1-1ubuntu3) ...
Selecting previously unselected package gettext.
Preparing to unpack .../gettext_0.18.3.1-1ubuntu3_armhf.deb ...
Unpacking gettext (0.18.3.1-1ubuntu3) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../intltool-debian_0.35.0+20060710.1_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.1) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../po-debconf_1.0.16+nmu2ubuntu1_all.deb ...
Unpacking po-debconf (1.0.16+nmu2ubuntu1) ...
Selecting previously unselected package dh-apparmor.
Preparing to unpack .../dh-apparmor_2.8.95~2430-0ubuntu5_all.deb ...
Unpacking dh-apparmor (2.8.95~2430-0ubuntu5) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../debhelper_9.20131227ubuntu1_all.deb ...
Unpacking debhelper (9.20131227ubuntu1) ...
Selecting previously unselected package libfakeroot:armhf.
Preparing to unpack .../libfakeroot_1.20-3ubuntu2_armhf.deb ...
Unpacking libfakeroot:armhf (1.20-3ubuntu2) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../fakeroot_1.20-3ubuntu2_armhf.deb ...
Unpacking fakeroot (1.20-3ubuntu2) ...
Selecting previously unselected package ruby1.9.1-dev.
Preparing to unpack .../ruby1.9.1-dev_1.9.3.484-2ubuntu1_armhf.deb ...
Unpacking ruby1.9.1-dev (1.9.3.484-2ubuntu1) ...
Selecting previously unselected package gem2deb.
Preparing to unpack .../archives/gem2deb_0.6.1_all.deb ...
Unpacking gem2deb (0.6.1) ...
Selecting previously unselected package rake.
Preparing to unpack .../archives/rake_10.0.4-1_all.deb ...
Unpacking rake (10.0.4-1) ...
Selecting previously unselected package rake-compiler.
Preparing to unpack .../rake-compiler_0.9.2-1_all.deb ...
Unpacking rake-compiler (0.9.2-1) ...
Selecting previously unselected package ruby-diff-lcs.
Preparing to unpack .../ruby-diff-lcs_1.2.4-1_all.deb ...
Unpacking ruby-diff-lcs (1.2.4-1) ...
Selecting previously unselected package ruby-rspec-expectations.
Preparing to unpack .../ruby-rspec-expectations_2.14.2-1_all.deb ...
Unpacking ruby-rspec-expectations (2.14.2-1) ...
Selecting previously unselected package ruby-rspec-mocks.
Preparing to unpack .../ruby-rspec-mocks_2.14.3-1_all.deb ...
Unpacking ruby-rspec-mocks (2.14.3-1) ...
Selecting previously unselected package ruby-rspec-core.
Preparing to unpack .../ruby-rspec-core_2.14.7-2_all.deb ...
Unpacking ruby-rspec-core (2.14.7-2) ...
Selecting previously unselected package ruby-rspec.
Preparing to unpack .../ruby-rspec_2.14.1-1_all.deb ...
Unpacking ruby-rspec (2.14.1-1) ...
Processing triggers for mime-support (3.54ubuntu1) ...
Setting up libasprintf0c2:armhf (0.18.3.1-1ubuntu3) ...
Setting up groff-base (1.22.2-5) ...
Setting up bsdmainutils (9.0.5ubuntu1) ...
update-alternatives: using /usr/bin/bsd-write to provide /usr/bin/write (write) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/write.1.gz because associated file /usr/share/man/man1/bsd-write.1.gz (of link group write) doesn't exist
update-alternatives: using /usr/bin/bsd-from to provide /usr/bin/from (from) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/from.1.gz because associated file /usr/share/man/man1/bsd-from.1.gz (of link group from) doesn't exist
Setting up man-db (2.6.7.1-1) ...
Building database of manual pages ...
Setting up libcroco3:armhf (0.6.8-2ubuntu1) ...
Setting up libunistring0:armhf (0.9.3-5ubuntu3) ...
Setting up gettext-base (0.18.3.1-1ubuntu3) ...
Setting up gettext (0.18.3.1-1ubuntu3) ...
Setting up intltool-debian (0.35.0+20060710.1) ...
Setting up po-debconf (1.0.16+nmu2ubuntu1) ...
Setting up dh-apparmor (2.8.95~2430-0ubuntu5) ...
Setting up debhelper (9.20131227ubuntu1) ...
Setting up libfakeroot:armhf (1.20-3ubuntu2) ...
Setting up fakeroot (1.20-3ubuntu2) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/fakeroot.1.gz because associated file /usr/share/man/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/faked.1.gz because associated file /usr/share/man/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/es/man1/fakeroot.1.gz because associated file /usr/share/man/es/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/es/man1/faked.1.gz because associated file /usr/share/man/es/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/fr/man1/fakeroot.1.gz because associated file /usr/share/man/fr/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/fr/man1/faked.1.gz because associated file /usr/share/man/fr/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/sv/man1/fakeroot.1.gz because associated file /usr/share/man/sv/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/sv/man1/faked.1.gz because associated file /usr/share/man/sv/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
Setting up ruby1.9.1-dev (1.9.3.484-2ubuntu1) ...
Setting up gem2deb (0.6.1) ...
Setting up rake (10.0.4-1) ...
Setting up rake-compiler (0.9.2-1) ...
Setting up ruby-diff-lcs (1.2.4-1) ...
Setting up ruby-rspec-expectations (2.14.2-1) ...
Setting up ruby-rspec-mocks (2.14.3-1) ...
Setting up ruby-rspec-core (2.14.7-2) ...
Setting up ruby-rspec (2.14.1-1) ...
Processing triggers for libc-bin (2.19-0ubuntu6) ...
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

user1@plum:~/ruby-iobuffer-1.1.2$

必要なパッケージを追加しましたので、再び debuild コマンドを実行します。

user1@plum:~/ruby-iobuffer-1.1.2$ sudo debuild -us -uc -b
 dpkg-buildpackage -rfakeroot -D -us -uc -b
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: source package ruby-iobuffer
dpkg-buildpackage: source version 1.1.2-2ppa1~trusty1
dpkg-buildpackage: source distribution trusty
dpkg-buildpackage: source changed by Mario Fetka <mario@debian7.disconnected-by-peer.at>
 dpkg-source --before-build ruby-iobuffer-1.1.2
dpkg-buildpackage: host architecture armhf
dpkg-source: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
 fakeroot debian/rules clean
dh clean --buildsystem=ruby --with ruby
   dh_testdir -O--buildsystem=ruby
   dh_auto_clean -O--buildsystem=ruby
/usr/lib/ruby/vendor_ruby/gem2deb.rb:17: warning: setting Encoding.default_external
〜〜〜 略 〜〜〜
dpkg-gencontrol: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe
dpkg-gencontrol: warning: package ruby-iobuffer: unused substitution variable ${misc:Pre-Depends}
   dh_md5sums -O--buildsystem=ruby
   dh_builddeb -O--buildsystem=ruby
dpkg-deb: warning: 'debian/ruby-iobuffer/DEBIAN/control' contains user-defined field 'Ruby-Versions'
dpkg-deb: warning: ignoring 1 warning about the control file(s)

dpkg-deb: building package `ruby-iobuffer' in `../ruby-iobuffer_1.1.2-2ppa1~trusty1_armhf.deb'.
 dpkg-genchanges -b >../ruby-iobuffer_1.1.2-2ppa1~trusty1_armhf.changes
dpkg-genchanges: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-genchanges: binary-only upload - not including any source code
 dpkg-source --after-build ruby-iobuffer-1.1.2
dpkg-source: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-buildpackage: binary only upload (no source included)
user1@plum:~/ruby-iobuffer-1.1.2$

パッケージが作成されました。

user1@plum:~/ruby-iobuffer-1.1.2$ cd ..
user1@plum:~$ ls -l ruby-iobuffer*deb
-rw-r--r-- 1 root root 7634 Jul 25 14:13 ruby-iobuffer_1.1.2-2ppa1~trusty1_armhf.deb
user1@plum:~$

ビルドされたパッケージをインストールしておきます。

user1@plum:~$ sudo dpkg -i ruby-iobuffer_1.1.2-2ppa1~trusty1_armhf.deb 
Selecting previously unselected package ruby-iobuffer.
(Reading database ... 25155 files and directories currently installed.)
Preparing to unpack ruby-iobuffer_1.1.2-2ppa1~trusty1_armhf.deb ...
Unpacking ruby-iobuffer (1.1.2-2ppa1~trusty1) ...
Setting up ruby-iobuffer (1.1.2-2ppa1~trusty1) ...
user1@plum:~$

ruby-cool.io のビルド

“ruby-iobuffer” と同様にして、ビルドを行います。

user1@plum:~$ sudo apt-get source ruby-cool.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Need to get 98.1 kB of source archives.
Get:1 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-cool.io 1.1.1-2ppa1~trusty1 (tar) [94.6 kB]
Get:2 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-cool.io 1.1.1-2ppa1~trusty1 (diff) [2214 B]
Get:3 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-cool.io 1.1.1-2ppa1~trusty1 (dsc) [1317 B]
Fetched 98.1 kB in 1s (49.9 kB/s) 
gpgv: Signature made Fri Apr 18 19:27:58 2014 JST using DSA key ID 0F0C7602
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./ruby-cool.io_1.1.1-2ppa1~trusty1.dsc
dpkg-source: info: extracting ruby-cool.io in ruby-cool.io-1.1.1
dpkg-source: info: unpacking ruby-cool.io_1.1.1.orig.tar.gz
dpkg-source: info: unpacking ruby-cool.io_1.1.1-2ppa1~trusty1.debian.tar.gz
user1@plum:~$

user1@plum:~$ cd ruby-cool.io-1.1.1/
user1@plum:~/ruby-cool.io-1.1.1$ sudo debuild -us -uc -b
 dpkg-buildpackage -rfakeroot -D -us -uc -b
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: source package ruby-cool.io
dpkg-buildpackage: source version 1.1.1-2ppa1~trusty1
dpkg-buildpackage: source distribution trusty
dpkg-buildpackage: source changed by Mario Fetka <mario@debian7.disconnected-by-peer.at>
 dpkg-source --before-build ruby-cool.io-1.1.1
dpkg-buildpackage: host architecture armhf
dpkg-source: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
 fakeroot debian/rules clean
dh clean --buildsystem=ruby --with ruby
   dh_testdir -O--buildsystem=ruby
〜〜〜 略 〜〜〜
dpkg-deb: warning: 'debian/ruby-cool.io/DEBIAN/control' contains user-defined field 'Ruby-Versions'
dpkg-deb: warning: ignoring 1 warning about the control file(s)

dpkg-deb: building package `ruby-cool.io' in `../ruby-cool.io_1.1.1-2ppa1~trusty1_armhf.deb'.
 dpkg-genchanges -b >../ruby-cool.io_1.1.1-2ppa1~trusty1_armhf.changes
dpkg-genchanges: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-genchanges: binary-only upload - not including any source code
 dpkg-source --after-build ruby-cool.io-1.1.1
dpkg-source: warning: unknown information field 'Dm-Upload-Allowed' in input data in general section of control info file
dpkg-buildpackage: binary only upload (no source included)
user1@plum:~/ruby-cool.io-1.1.1$

こちらもパッケージが作成されました。

user1@plum:~/ruby-cool.io-1.1.1$ cd ..
user1@plum:~$ ls -l ruby-cool.io*deb
-rw-r--r-- 1 root root 47100 Jul 25 14:37 ruby-cool.io_1.1.1-2ppa1~trusty1_armhf.deb
user1@plum:~$

同様に、ビルドされたパッケージをインストールしておきます。

user1@plum:~$ sudo dpkg -i ruby-cool.io_1.1.1-2ppa1~trusty1_armhf.deb
Selecting previously unselected package ruby-cool.io.
(Reading database ... 25167 files and directories currently installed.)
Preparing to unpack ruby-cool.io_1.1.1-2ppa1~trusty1_armhf.deb ...
Unpacking ruby-cool.io (1.1.1-2ppa1~trusty1) ...
Setting up ruby-cool.io (1.1.1-2ppa1~trusty1) ...
user1@plum:~$

ruby-fluentd のインストール

以上で、“ruby-fluentd” が依存する(バイナリパッケージが用意されていない)パッケージのビルド・インストールができましたので、
ruby-fluentd をインストールします。

user1@plum:~$ sudo apt-get install ruby-fluentd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libhttp-parser2.1 libjemalloc1 ruby-http-parser.rb ruby-json ruby-msgpack
  ruby-sigdump ruby-yajl
Suggested packages:
  ruby-http-parser.rb-doc
The following NEW packages will be installed:
  libhttp-parser2.1 libjemalloc1 ruby-fluentd ruby-http-parser.rb ruby-json
  ruby-msgpack ruby-sigdump ruby-yajl
0 upgraded, 8 newly installed, 0 to remove and 2 not upgraded.
Need to get 254 kB of archives.
After this operation, 1533 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-sigdump all 0.2.2-2ppa1~trusty1 [3784 B]
Get:2 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe libhttp-parser2.1 armhf 2.1-2 [17.3 kB]
Get:3 http://ppa.launchpad.net/mario-fetka/fluentd/ubuntu/ trusty/main ruby-fluentd all 0.10.45-2ppa1~trusty1 [50.9 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe libjemalloc1 armhf 3.5.1-2 [70.6 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ruby-msgpack armhf 0.5.5-2 [25.6 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ruby-json armhf 1.8.0-1build1 [44.0 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ruby-yajl armhf 1.2.0-1 [32.0 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ruby-http-parser.rb armhf 0.6.0~beta.2-1 [10.1 kB]
Fetched 254 kB in 3s (83.4 kB/s)          
Selecting previously unselected package libhttp-parser2.1:armhf.
(Reading database ... 25193 files and directories currently installed.)
Preparing to unpack .../libhttp-parser2.1_2.1-2_armhf.deb ...
Unpacking libhttp-parser2.1:armhf (2.1-2) ...
Selecting previously unselected package libjemalloc1.
Preparing to unpack .../libjemalloc1_3.5.1-2_armhf.deb ...
Unpacking libjemalloc1 (3.5.1-2) ...
Selecting previously unselected package ruby-msgpack.
Preparing to unpack .../ruby-msgpack_0.5.5-2_armhf.deb ...
Unpacking ruby-msgpack (0.5.5-2) ...
Selecting previously unselected package ruby-json.
Preparing to unpack .../ruby-json_1.8.0-1build1_armhf.deb ...
Unpacking ruby-json (1.8.0-1build1) ...
Selecting previously unselected package ruby-yajl.
Preparing to unpack .../ruby-yajl_1.2.0-1_armhf.deb ...
Unpacking ruby-yajl (1.2.0-1) ...
Selecting previously unselected package ruby-http-parser.rb.
Preparing to unpack .../ruby-http-parser.rb_0.6.0~beta.2-1_armhf.deb ...
Unpacking ruby-http-parser.rb (0.6.0~beta.2-1) ...
Selecting previously unselected package ruby-sigdump.
Preparing to unpack .../ruby-sigdump_0.2.2-2ppa1~trusty1_all.deb ...
Unpacking ruby-sigdump (0.2.2-2ppa1~trusty1) ...
Selecting previously unselected package ruby-fluentd.
Preparing to unpack .../ruby-fluentd_0.10.45-2ppa1~trusty1_all.deb ...
Unpacking ruby-fluentd (0.10.45-2ppa1~trusty1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libhttp-parser2.1:armhf (2.1-2) ...
Setting up libjemalloc1 (3.5.1-2) ...
Setting up ruby-msgpack (0.5.5-2) ...
Setting up ruby-json (1.8.0-1build1) ...
Setting up ruby-yajl (1.2.0-1) ...
Setting up ruby-http-parser.rb (0.6.0~beta.2-1) ...
Setting up ruby-sigdump (0.2.2-2ppa1~trusty1) ...
Setting up ruby-fluentd (0.10.45-2ppa1~trusty1) ...
Creating system group: fluent
Adding group `fluent' (GID 112) ...
Done.
Creating user fluent in group fluent
 * Starting fluent  fluent                                                                                    2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:214:rescue in main_process: unexpected error error="No such file or directory - /etc/fluent/fluent.conf"
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:276:in `read'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:276:in `read_config'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:88:in `block in start'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:198:in `call'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:198:in `main_process'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:173:in `block in supervise'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:172:in `fork'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:172:in `supervise'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/supervisor.rb:87:in `start'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/vendor_ruby/fluent/command/fluentd.rb:168:in `<top (required)>'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:53:in `require'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:53:in `require'
  2014-07-25 14:43:38 +0900 [error]: fluent/supervisor.rb:172:fork: /usr/bin/fluentd:6:in `<main>'
                                                                                                       [fail]
Processing triggers for libc-bin (2.19-0ubuntu6) ...
Processing triggers for ureadahead (0.100.0-16) ...
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB

Total disk space freed by localepurge: 0 KiB

user1@plum:~$

セットアップ後、自動で起動処理が行われてしまいました。
設定ファイル /etc/fluent/fluent.conf が無いので起動は失敗したようです。

動作テスト

簡単な設定ファイルを作り、基本的な動作確認をしてみます。

user1@plum:~$ sudo mkdir /etc/fluent
user1@plum:~$ sudo nano -w /etc/fluent/fluent.conf

fluent.conf

<source>
  type forward
</source>
<match debug.*>
  type file
  path /tmp/fluent_debug.txt
</match>

標準入力から入ってきたものの中で、タグが “debug.*” にマッチしたものを /tmp/fluent_debug.txt へ出力する設定です。

fluentd を起動します。

user1@plum:~$ sudo service ruby-fluentd start
 * Starting fluent  fluent                                                                             [ OK ] 
user1@plum:~$

単純なデータを入力してみます。タグは “debug.test” にします。

user1@plum:~$ echo '{"key" : "value"}' | fluent-cat debug.test

出力されたか確認してみます。

user1@plum:~$ ls -l /tmp/fluent_debug*                               
-rw-r--r-- 1 fluent fluent 109 Jul 25 15:09 /tmp/fluent_debug.txt.20140725.b4fefe4bdbaa9b437
user1@plum:~$ cat /tmp/fluent_debug.txt.20140725.b4fefe4bdbaa9b437 
2014-07-25T15:09:41+09:00	debug.test	{"key":"value"}

処理が行われたのが確認できました。

サーバの設定

コンテナの作成

クライアント(MA-E3xx)側の Fluentd はとりあえずセットアップできましたので、データを蓄積・可視化するサーバ側もセットアップします。
今回は、サーバには LXC⁴⁾ によるコンテナ型仮想環境を準備します。

Fluentd の公式パッケージがサポートしているのが Ubuntu 12.04LTS (Precise) なので、Ubuntu 12.04LTS の環境を作成します。
参照: http://docs.fluentd.org/articles/install-by-deb

user1@ubuntu:~$ sudo lxc-create -t ubuntu -n precise64 -- -r precise

Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... 
Installing packages in template: ssh,vim,language-pack-en,language-pack-ja
Downloading ubuntu precise minimal ...
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
I: Valid Release signature (key id 630239CC130E1A7FD81A27B140976EAF437D05B5)
〜〜〜 略 〜〜〜
Generating locales...
  en_US.UTF-8... done
Generation complete.
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...

Current default time zone: 'Asia/Tokyo'
Local time is now:      Fri Aug  1 16:20:33 JST 2014.
Universal Time is now:  Fri Aug  1 07:20:33 UTC 2014.


##
# The default user is 'ubuntu' with password 'ubuntu'!
# Use the 'sudo' command to run tasks as root in the container.
##

user1@ubuntu:~$

コンテナの起動 / 設定

作成したコンテナを起動します。
まだネットワークの設定を行っていませんので、フォアグランドで起動させます。

user1@ubuntu:~$ sudo lxc-start -n precise64

Ubuntu 12.04.4 LTS precise64 console

precise64 login: <4>init: setvtrgb main process (272) terminated with status 1


Ubuntu 12.04.4 LTS precise64 console

precise64 login: ubuntu
Password: 
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.13.0-32-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@precise64:~$

ログインできました。

使用する環境に合わせて、

ネットワーク設定
ユーザアカウント追加

などを行っておきます。ubuntu ユーザも削除しておくほうがよいと思います。

設定が完了したら一旦コンテナを終了し、再度 -d オプション (daemonize the container) 付きで起動します。

user1@ubuntu:~$ sudo lxc-start -n precise64 -d

今度は SSH でログインします。

user1@ubuntu:~$ ssh -l user1 192.168.103.192
user1@192.168.103.192's password: 
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.13.0-32-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Fri Aug  1 16:31:33 2014
user1@precise64:~$

td-agent のセットアップ

公式サイトの説明のとおり作業します。
※ curl は初期状態ではインストールされていないため、apt-get でインストールが必要です。

user1@precise64:~$ curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   555  100   555    0     0   3037      0 --:--:-- --:--:-- --:--:-- 92500
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
[sudo] password for user1: 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   934  100   934    0     0   4330      0 --:--:-- --:--:-- --:--:--  130k
OK
Get:1 http://packages.treasuredata.com precise Release.gpg [198 B]
Get:2 http://packages.treasuredata.com precise Release [2,377 B]                                           
Get:3 http://packages.treasuredata.com precise/contrib amd64 Packages [447 B]                           
Get:4 http://packages.treasuredata.com precise/contrib i386 Packages [451 B]                                 
Ign http://packages.treasuredata.com precise/contrib TranslationIndex                                        
Hit http://archive.ubuntu.com precise Release.gpg                                                            
Hit http://archive.ubuntu.com precise-updates Release.gpg                                                  
Hit http://security.ubuntu.com precise-security Release.gpg           
Hit http://archive.ubuntu.com precise Release   
Hit http://security.ubuntu.com precise-security Release                                      
Hit http://archive.ubuntu.com precise-updates Release                  
Hit http://security.ubuntu.com precise-security/main amd64 Packages   
Hit http://archive.ubuntu.com precise/main amd64 Packages            
Hit http://archive.ubuntu.com precise/restricted amd64 Packages      
Hit http://archive.ubuntu.com precise/universe amd64 Packages        
Hit http://archive.ubuntu.com precise/multiverse amd64 Packages      
Hit http://archive.ubuntu.com precise/main i386 Packages             
Hit http://archive.ubuntu.com precise/restricted i386 Packages       
Hit http://archive.ubuntu.com precise/universe i386 Packages         
Hit http://archive.ubuntu.com precise/multiverse i386 Packages       
Hit http://archive.ubuntu.com precise/main TranslationIndex          
Hit http://archive.ubuntu.com precise/multiverse TranslationIndex
Hit http://security.ubuntu.com precise-security/restricted amd64 Packages
Hit http://security.ubuntu.com precise-security/universe amd64 Packages
Hit http://security.ubuntu.com precise-security/multiverse amd64 Packages
Hit http://security.ubuntu.com precise-security/main i386 Packages   
Hit http://security.ubuntu.com precise-security/restricted i386 Packages
Hit http://security.ubuntu.com precise-security/universe i386 Packages
Hit http://security.ubuntu.com precise-security/multiverse i386 Packages
Hit http://security.ubuntu.com precise-security/main TranslationIndex
Hit http://security.ubuntu.com precise-security/multiverse TranslationIndex
Hit http://security.ubuntu.com precise-security/restricted TranslationIndex
Hit http://archive.ubuntu.com precise/restricted TranslationIndex    
Hit http://archive.ubuntu.com precise/universe TranslationIndex      
Hit http://archive.ubuntu.com precise-updates/main amd64 Packages    
Hit http://archive.ubuntu.com precise-updates/restricted amd64 Packages
Hit http://archive.ubuntu.com precise-updates/universe amd64 Packages
Hit http://archive.ubuntu.com precise-updates/multiverse amd64 Packages
Hit http://archive.ubuntu.com precise-updates/main i386 Packages     
Ign http://packages.treasuredata.com precise/contrib Translation-en_US
Hit http://security.ubuntu.com precise-security/universe TranslationIndex
Ign http://packages.treasuredata.com precise/contrib Translation-en   
Hit http://archive.ubuntu.com precise-updates/restricted i386 Packages
Hit http://archive.ubuntu.com precise-updates/universe i386 Packages
Hit http://archive.ubuntu.com precise-updates/multiverse i386 Packages
Hit http://archive.ubuntu.com precise-updates/main TranslationIndex
Hit http://archive.ubuntu.com precise-updates/multiverse TranslationIndex
Hit http://archive.ubuntu.com precise-updates/restricted TranslationIndex
Hit http://archive.ubuntu.com precise-updates/universe TranslationIndex
Hit http://archive.ubuntu.com precise/main Translation-en
Hit http://security.ubuntu.com precise-security/main Translation-en
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Hit http://archive.ubuntu.com precise/multiverse Translation-en
Hit http://archive.ubuntu.com precise/restricted Translation-en
Hit http://security.ubuntu.com precise-security/restricted Translation-en
Hit http://archive.ubuntu.com precise/universe Translation-en
Hit http://archive.ubuntu.com precise-updates/main Translation-en
Hit http://archive.ubuntu.com precise-updates/multiverse Translation-en
Hit http://archive.ubuntu.com precise-updates/restricted Translation-en
Hit http://archive.ubuntu.com precise-updates/universe Translation-en
Hit http://security.ubuntu.com precise-security/universe Translation-en
Fetched 3,473 B in 2s (1,383 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libcap2 libclass-isa-perl libgdbm3 libopts25 libssl0.9.8 libswitch-perl libxml2 libxslt1.1 libyaml-0-2 ntp
  perl perl-modules sgml-base xml-core
Suggested packages:
  ntp-doc apparmor perl-doc libterm-readline-gnu-perl libterm-readline-perl-perl make libpod-plainer-perl
  sgml-base-doc debhelper
The following NEW packages will be installed:
  libcap2 libclass-isa-perl libgdbm3 libopts25 libssl0.9.8 libswitch-perl libxml2 libxslt1.1 libyaml-0-2 ntp
  perl perl-modules sgml-base td-agent xml-core
0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
Need to get 71.3 MB of archives.
After this operation, 136 MB of additional disk space will be used.
Get:1 http://packages.treasuredata.com/precise/ precise/contrib td-agent amd64 1.1.20-1 [61.0 MB]
Get:2 http://archive.ubuntu.com/ubuntu/ precise-updates/universe libssl0.9.8 amd64 0.9.8o-7ubuntu3.2 [868 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ precise/main libgdbm3 amd64 1.8.3-10 [35.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ precise-updates/main libxml2 amd64 2.7.8.dfsg-5.1ubuntu4.9 [673 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ precise/main libcap2 amd64 1:2.22-1ubuntu3 [12.0 kB]
Get:6 http://archive.ubuntu.com/ubuntu/ precise-updates/main libxslt1.1 amd64 1.1.26-8ubuntu1.3 [167 kB]
Get:7 http://archive.ubuntu.com/ubuntu/ precise-updates/main libyaml-0-2 amd64 0.1.4-2ubuntu0.12.04.3 [57.8 kB]
Get:8 http://archive.ubuntu.com/ubuntu/ precise/main libopts25 amd64 1:5.12-0.1ubuntu1 [59.9 kB]
Get:9 http://archive.ubuntu.com/ubuntu/ precise-updates/main ntp amd64 1:4.2.6.p3+dfsg-1ubuntu3.1 [612 kB]
Get:10 http://archive.ubuntu.com/ubuntu/ precise/main sgml-base all 1.26+nmu1ubuntu1 [8,360 B]
Get:11 http://archive.ubuntu.com/ubuntu/ precise/main xml-core all 0.13 [23.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu/ precise/main libclass-isa-perl all 0.36-3 [11.9 kB]
Get:13 http://archive.ubuntu.com/ubuntu/ precise-updates/main perl-modules all 5.14.2-6ubuntu2.4 [3,389 kB]
Get:14 http://archive.ubuntu.com/ubuntu/ precise-updates/main perl amd64 5.14.2-6ubuntu2.4 [4,411 kB]
Get:15 http://archive.ubuntu.com/ubuntu/ precise/main libswitch-perl all 2.16-2 [19.2 kB]
Fetched 71.3 MB in 6s (10.8 MB/s)                                                                            
Preconfiguring packages ...
Selecting previously unselected package libssl0.9.8.
(Reading database ... 12876 files and directories currently installed.)
Unpacking libssl0.9.8 (from .../libssl0.9.8_0.9.8o-7ubuntu3.2_amd64.deb) ...
Selecting previously unselected package libgdbm3.
Unpacking libgdbm3 (from .../libgdbm3_1.8.3-10_amd64.deb) ...
Selecting previously unselected package libxml2.
Unpacking libxml2 (from .../libxml2_2.7.8.dfsg-5.1ubuntu4.9_amd64.deb) ...
Selecting previously unselected package libcap2.
Unpacking libcap2 (from .../libcap2_1%3a2.22-1ubuntu3_amd64.deb) ...
Selecting previously unselected package libxslt1.1.
Unpacking libxslt1.1 (from .../libxslt1.1_1.1.26-8ubuntu1.3_amd64.deb) ...
Selecting previously unselected package libyaml-0-2.
Unpacking libyaml-0-2 (from .../libyaml-0-2_0.1.4-2ubuntu0.12.04.3_amd64.deb) ...
Selecting previously unselected package libopts25.
Unpacking libopts25 (from .../libopts25_1%3a5.12-0.1ubuntu1_amd64.deb) ...
Selecting previously unselected package ntp.
Unpacking ntp (from .../ntp_1%3a4.2.6.p3+dfsg-1ubuntu3.1_amd64.deb) ...
Selecting previously unselected package sgml-base.
Unpacking sgml-base (from .../sgml-base_1.26+nmu1ubuntu1_all.deb) ...
Selecting previously unselected package xml-core.
Unpacking xml-core (from .../archives/xml-core_0.13_all.deb) ...
Selecting previously unselected package libclass-isa-perl.
Unpacking libclass-isa-perl (from .../libclass-isa-perl_0.36-3_all.deb) ...
Selecting previously unselected package perl-modules.
Unpacking perl-modules (from .../perl-modules_5.14.2-6ubuntu2.4_all.deb) ...
Selecting previously unselected package perl.
Unpacking perl (from .../perl_5.14.2-6ubuntu2.4_amd64.deb) ...
Selecting previously unselected package libswitch-perl.
Unpacking libswitch-perl (from .../libswitch-perl_2.16-2_all.deb) ...
Selecting previously unselected package td-agent.
Unpacking td-agent (from .../td-agent_1.1.20-1_amd64.deb) ...
Processing triggers for ureadahead ...
Setting up libssl0.9.8 (0.9.8o-7ubuntu3.2) ...
Setting up libgdbm3 (1.8.3-10) ...
Setting up libxml2 (2.7.8.dfsg-5.1ubuntu4.9) ...
Setting up libcap2 (1:2.22-1ubuntu3) ...
Setting up libxslt1.1 (1.1.26-8ubuntu1.3) ...
Setting up libyaml-0-2 (0.1.4-2ubuntu0.12.04.3) ...
Setting up libopts25 (1:5.12-0.1ubuntu1) ...
Setting up ntp (1:4.2.6.p3+dfsg-1ubuntu3.1) ...
 * Starting NTP server ntpd                                                                            [ OK ] 
Setting up sgml-base (1.26+nmu1ubuntu1) ...
Setting up xml-core (0.13) ...
Setting up libclass-isa-perl (0.36-3) ...
Setting up td-agent (1.1.20-1) ...
Adding system user `td-agent' (UID 104) ...
Adding new group `td-agent' (GID 106) ...
Adding new user `td-agent' (UID 104) with group `td-agent' ...
Not creating home directory `/home/td-agent'.
Installing default conffile /etc/td-agent/td-agent.conf ...
 * Starting td-agent  td-agent                                                                         [ OK ] 
Setting up libswitch-perl (2.16-2) ...
Setting up perl-modules (5.14.2-6ubuntu2.4) ...
Setting up perl (5.14.2-6ubuntu2.4) ...
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode.
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
user1@precise64:~$

起動しているか、確認します。

user1@precise64:~$ ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 /sbin/init
  114 ?        S      0:00 upstart-udev-bridge --daemon
  145 ?        Ss     0:00 /sbin/udevd --daemon
  153 ?        Sl     0:00 rsyslogd -c5
  155 ?        S      0:00 upstart-socket-bridge --daemon
  203 ?        Ss     0:00 /usr/sbin/sshd -D
  231 ?        Ss+    0:00 /sbin/getty -8 38400 tty4
  234 ?        Ss+    0:00 /sbin/getty -8 38400 tty2
  235 ?        Ss+    0:00 /sbin/getty -8 38400 tty3
  241 ?        Ss     0:00 cron
  268 ?        Ss+    0:00 /sbin/getty -8 38400 console
  271 ?        Ss+    0:00 /sbin/getty -8 38400 tty1
  303 ?        Ss     0:00 sshd: user1 [priv]  
  315 ?        S      0:00 sshd: user1@pts/0   
  316 pts/0    Ss     0:00 -bash
 3609 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 103:105
 3732 ?        Sl     0:00 /usr/lib/fluent/ruby/bin/ruby /usr/sbin/td-agent --daemon /var/run/td-agent/td-agent.pid --log /var/log/td-agent/td-agent.log
 3735 ?        Sl     0:00 /usr/lib/fluent/ruby/bin/ruby /usr/sbin/td-agent --daemon /var/run/td-agent/td-agent.pid --log /var/log/td-agent/td-agent.log
 3757 pts/0    R+     0:00 ps ax
user1@precise64:~$

Java のセットアップ

elasticsearch は JVM 上で動作しますので、Java をインストールしておきます。
参照: Install Oracle Java 7 in Ubuntu via PPA Repository

準備

user1@precise64:~$ sudo apt-get install python-software-properties

user1@precise64:~$ sudo add-apt-repository ppa:webupd8team/java
You are about to add the following PPA to your system:
 Oracle Java (JDK) Installer (automatically downloads and installs Oracle JDK6 / JDK7 / JDK8). There are no actual Java files in this PPA. 

More info: 
- for Oracle Java 7: http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html
- for Oracle Java 8: http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

Debian installation instructions: http://www.webupd8.org/2012/06/how-to-install-oracle-java-7-in-debian.html
 More info: https://launchpad.net/~webupd8team/+archive/ubuntu/java
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpFd_z7_/secring.gpg' created
gpg: keyring `/tmp/tmpFd_z7_/pubring.gpg' created
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpFd_z7_/trustdb.gpg: trustdb created
gpg: key EEA14886: public key "Launchpad VLC" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
user1@precise64:~$

user1@precise64:~$ sudo apt-get install oracle-java7-installer
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  binutils gsfonts gsfonts-x11 java-common libfontenc1 libfreetype6 libxfont1 wget x11-common
  xfonts-encodings xfonts-utils
Suggested packages:
  binutils-doc default-jre equivs binfmt-support visualvm ttf-baekmuk ttf-unfonts ttf-unfonts-core
  ttf-kochi-gothic ttf-sazanami-gothic ttf-kochi-mincho ttf-sazanami-mincho ttf-arphic-uming firefox
  firefox-2 iceweasel mozilla-firefox iceape-browser mozilla-browser epiphany-gecko epiphany-webkit
  epiphany-browser galeon midbrowser moblin-web-browser xulrunner xulrunner-1.9 konqueror chromium-browser
  midori google-chrome
The following NEW packages will be installed:
  binutils gsfonts gsfonts-x11 java-common libfontenc1 libfreetype6 libxfont1 oracle-java7-installer wget
  x11-common xfonts-encodings xfonts-utils
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 7,609 kB of archives.
After this operation, 17.8 MB of additional disk space will be used.
Do you want to continue [Y/n]?

セットアップされました。

user1@precise64:~$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
user1@precise64:~$

elasticsearch のセットアップ

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html の説明のとおり、セットアップします。

user1@precise64:~$ wget -qO - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
OK
user1@precise64:~$

リポジトリを追加します。

user1@precise64:~$ sudo nano -w /etc/apt/sources.list

sources.list

deb http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu precise-security main restricted universe multiverse
 
deb http://packages.elasticsearch.org/elasticsearch/1.3/debian stable main

elasticsearch をインストールします。

user1@precise64:~$ sudo apt-get update

user1@precise64:~$ sudo apt-get install elasticsearch
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 26.9 MB of archives.
After this operation, 30.9 MB of additional disk space will be used.
Get:1 http://packages.elasticsearch.org/elasticsearch/1.3/debian/ stable/main elasticsearch all 1.3.1 [26.9 MB]
Fetched 26.9 MB in 5s (4,866 kB/s)        
Selecting previously unselected package elasticsearch.
(Reading database ... 20533 files and directories currently installed.)
Unpacking elasticsearch (from .../elasticsearch_1.3.1_all.deb) ...
Processing triggers for ureadahead ...
Setting up elasticsearch (1.3.1) ...
Adding system user `elasticsearch' (UID 105) ...
Adding new user `elasticsearch' (UID 105) with group `elasticsearch' ...
Not creating home directory `/usr/share/elasticsearch'.
### NOT starting elasticsearch by default on bootup, please execute
 sudo update-rc.d elasticsearch defaults 95 10
### In order to start elasticsearch, execute
 sudo /etc/init.d/elasticsearch start
user1@precise64:~$

fluentd-ui のセットアップ

¹⁾

参照: 開発環境のセットアップ(セルフ)

²⁾

参照: SoftEther の導入

³⁾

フィボナッチ数列でベンチマークの結果の通り、Ruby1.9系より高速なためです。

⁴⁾

docker による方法も、のちほど試そうと考えています

MA-X/MA-S/MA-E/IP-K Developers' WiKi

目次

fluentd + elasticsearch + Kibana3 によるログ収集・可視化

参考

書籍

Web

準備

Ruby 開発環境のセットアップ

パッケージの作成

PPA の追加

パッケージの確認

パッケージのビルド

ruby-iobuffer のビルド

ruby-cool.io のビルド

ruby-fluentd のインストール

動作テスト

サーバの設定

コンテナの作成

コンテナの起動 / 設定

td-agent のセットアップ

Java のセットアップ

elasticsearch のセットアップ

fluentd-ui のセットアップ

MA-X/MA-S/MA-E/IP-K Developers' WiKi

ユーザ用ツール

サイト用ツール

目次

fluentd + elasticsearch + Kibana3 によるログ収集・可視化

参考

書籍

Web

準備

Ruby 開発環境のセットアップ

パッケージの作成

PPA の追加

パッケージの確認

パッケージのビルド

ruby-iobuffer のビルド

ruby-cool.io のビルド

ruby-fluentd のインストール

動作テスト

サーバの設定

コンテナの作成

コンテナの起動 / 設定

td-agent のセットアップ

Java のセットアップ

elasticsearch のセットアップ

fluentd-ui のセットアップ

ページ用ツール