Tsar

Introduction

Tsar (Taobao System Activity Reporter) is a monitoring tool, which can be used to gather and summarize system information, e.g. CPU, load, IO, and application information, e.g. nginx, HAProxy, Squid, etc. The results can be stored at local disk or sent to Nagios.

Tsar can be easily extended by writing modules, which makes it a powerful and versatile reporting tool.
Installation

Tsar is available on GitHub, you can clone and install it as follows:

$ git clone git://github.com/kongjian/tsar.git
$ cd tsar
$ make
# make install

Or you can download the zip file and install it:

$ wget -O tsar.zip https://github.com/alibaba/tsar/archive/master.zip --no-check-certificate
$ unzip tsar.zip
$ cd tsar
$ make
# make install

After installation, you may see these files:

    /etc/tsar/tsar.conf, which is tsar's main configuration file;
    /etc/cron.d/tsar, is used to run tsar to collect information every minute;
    /etc/logrotate.d/tsar will rotate tsar's log files every month;
    /usr/local/tsar/modules is the directory where all module libraries (*.so) are located;

Configuration

There is no output displayed after installation by default. Just run tsar -l to see if the real-time monitoring works, for instance:

[kongjian@tsar]$ tsar -l -i 1
Time              ---cpu-- ---mem-- ---tcp-- -----traffic---- --xvda-- -xvda1-- -xvda2-- -xvda3-- -xvda4-- -xvda5--  ---load-
Time                util     util   retran    pktin  pktout     util     util     util     util     util     util     load1
11/04/13-14:09:10   0.20    11.57     0.00     9.00    2.00     0.00     0.00     0.00     0.00     0.00     0.00      0.00
11/04/13-14:09:11   0.20    11.57     0.00     4.00    2.00     0.00     0.00     0.00     0.00     0.00     0.00      0.00

Usually, we configure Tsar by simply editing /etc/tsar/tsar.conf:

    To add a module, add a line like mod_<yourmodname> on
    To enable or disable a module, use mod_<yourmodname> on/off
    To specify parameters for a module, use mod_<yourmodname> on parameter
    output_stdio_mod is to set modules output to standard I/O
    output_file_path is to set history data file, (you should modify the logrotate script /etc/logrotate.d/tsar too)
    output_interface specifies tsar data output destination, which by default is a local file. See the Advanced section for more information.

Usage

    null :see default mods history data, tsar
    --modname :specify module to show, tsar --cpu
    -L/--list :list available moudule, tsar -L
    -l/--live :show real-time info, tsar -l --cpu
    -i/--interval :set interval for report, tsar -i 1 --cpu
    -s/--spec :specify module detail field, tsar --cpu -s sys,util
    -D/--detail :do not conver data to K/M/G, tsar --mem -D
    -m/--merge :merge multiply item to one, tsar --io -m
    -I/--item :show spec item data, tsar --io -I sda
    -d/--date :specify data, YYYYMMDD, or n means n days ago
    -C/--check :show the last collect data
    -h/--help :show help, tsar -h

Advanced

    Output to Nagios

To turn it on, just set output type output_interface file,nagios in the main configuration file.

You should also specify Nagios' IP address, port, and sending interval, e.g.:

####The IP address or the hostname running the NSCA daemon
server_addr nagios.server.com
####The port on which the daemon is listening - by default it is 5667
server_port 8086
####The cycle (interval) of sending alerts to Nagios
cycle_time 300

As tsar uses Nagios' passive mode, so you should specify the nsca binary and its configuration file, e.g.:

####nsca client program
send_nsca_cmd /usr/bin/send_nsca
send_nsca_conf /home/a/conf/amon/send_nsca.conf

Then specify the module and fields to be checked. There are 4 threshold levels.

####tsar mod alert config file
####threshold servicename.key;w-min;w-max;c-min;cmax;
threshold cpu.util;50;60;70;80;

    Output to MySQL

To use this feature, just add output type output_interface file,db in tsar's configuration file.

Then specify which module(s) will be enabled:

output_db_mod mod_cpu,mod_mem,mod_traffic,mod_load,mod_tcp,mod_udpmod_io

Note that you should set the IP address (or hostname) and port where tsar2db listens, e.g.:

output_db_addr console2:56677

Tsar2db receives sql data and flush it to MySQL. You can find more information about tsar2db at https://github.com/alibaba/tsar2db.
Module development

Tsar is easily extended. Whenever you want information that is not collected by tsar yet, you can write a module.

First, install the tsardevel tool (make tsardevel will do this for you):

Then run tsardevel <yourmodname>, and you will get a directory named yourmodname, e.g.:

[kongjian@tsar]$ tsardevel test
build:make
install:make install
uninstall:make uninstall

[kongjian@tsar]$ ls test
Makefile  mod_test.c  mod_test.conf

You can modify the cread_test_stats() and set_test_record() functions in test.c as you need. Then run make;make install to install your module and run tsar --yourmodname to see the output.
More

The homepage of Tsar is at http://tsar.taobao.org

You can also send your questions to kongjian@taobao.com if you have.

dstat命令

dstat是一个用来替换 vmstat, iostat, netstat, nfsstat 和 ifstat 这些命令的工具,是一个全能系统信息统计工具。

下载链接http://pkgs.repoforge.org/dstat/dstat-0.7.2-1.el5.rfx.noarch.rpm

安装 rpm -ivh dstat-0.7.2-1.el5.rfx.noarch.rpm 或者 yum install dstat

[root@monitor tool]# dstat 
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
 10   3  86   0   0   0|3026B 3286k|   0     0 |   0     0 | 531  2275 
 10   3  88   0   0   0|   0     0 |1059B 1538B|   0     0 |1015   774 
 10   2  87   0   0   0|   0     0 | 386B  815B|   0     0 |1033   861 
 10   2  87   0   0   0|  16k 6272k|1288B 2005B|   0     0 |1141   854 
[root@monitor tool]# dstat -lasmt
---load-avg--- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ----swap--- ------memory-usage----- -----time-----
 1m   5m  15m |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw | used  free| used  buff  cach  free|  date/time   
 1.2  0.8  0.6| 10   3  86   0   0   0|3026B 3286k|   0     0 |   0     0 | 531  2275 | 132k   18G|  10G  232M 4625M  442M|22-03 11:01:43
 1.2  0.8  0.6| 22   7  71   0   0   0|   0     0 |  39k   25k|   0     0 |1421  2913 | 132k   18G|  10G  232M 4626M  439M|22-03 11:01:44
 1.1  0.8  0.6|  1   0  99   0   0   0|   0     0 |  86k   20k|   0     0 |1036   393 | 132k   18G|  10G  232M 4626M  468M|22-03 11:01:45
 1.1  0.8  0.6|  0   0  99   0   0   0|   0  5704k|1288B 1796B|   0     0 |1175   418 | 132k   18G|  10G  232M 4626M  481M|22-03 11:01:46
[root@monitor tool]#  dstat --top-bio --top-io --top-cpu --top-mem
----most-expensive---- ----most-expensive---- -most-expensive- --most-expensive-
  block i/o process   |     i/o process      |  cpu process   |  memory process 
init [3]    241k  331k|init [3]   9088k 1611k|kjournald    0.0|mysqld      35.0M
                      |mysqld     5187k   59k|httpd        0.9|mysqld      35.0M
                      |top          64k 9027B|top          0.1|mysqld      35.0M
                      |mysqld      436B 2740B|                |mysqld      35.0M
kjournald     0    64k|httpd       158k   23k|httpd        0.1|mysqld      35.0M
kjournald     0   128k|httpd       165k   87k|httpd        0.2|mysqld      35.0M
                      |mysqld     8770k   93B|mysqld       2.1|mysqld      39.0M
httpd         0    48k|mysqld       20M  864k|httpd        0.9|mysqld      35.0M
                      |top          64k 9020B|top          0.1|mysqld      35.0M
kjournald     0    12k|httpd       838B    0 |                |mysqld      35.0M
kjournald     0    28k|snmpd      2817B    0 |                |mysqld      35.0M
                      |top          65k 9027B|                |mysqld      35.0M
httpd         0    24k|mysqld     5163k   59k|httpd        1.1|mysqld      35.0M
mysqld     4096B    0 |mysqld     5255k   53k|httpd        0.7|mysqld      35.0M
                      |top          65k 9027B|top          0.1|mysqld      35.0M
[root@monitor tool]# dstat --top-bio --top-io --top-cpu --top-mem --disk-tps --disk-util 
----most-expensive---- ----most-expensive---- -most-expensive- --most-expensive- -dsk/total- sda--sdb-
  block i/o process   |     i/o process      |  cpu process   |  memory process |reads writs|util:util
init [3]    241k  331k|init [3]   9088k 1611k|kjournald    0.0|mysqld      35.0M|   7    43 |0.14:0.76
php           0  4096B|mysqld      215M    0 |mysqld       6.2|mysqld      35.0M|   0     0 |   0:   0
mysqld     4096B  100k|mysqld       97M   21k|mysqld       3.4|mysqld      40.0M|   1     3 |   0:2.50
mysqld     4096B  124k|mysqld      478M  107k|httpd        0.2|mysqld      35.0M|   0    47 |   0:18.4
httpd         0    24k|httpd       165k   25k|httpd        0.1|mysqld      35.0M|   0     0 |   0:   0
kjournald     0   128k|snmpd      1524B  241B|syslogd      0.1|mysqld      35.0M|   0    18 |   0:7.80
httpd         0  4096B|init [3]     11k  652B|init [3]     0.1|mysqld      35.0M|   0     0 |   0:   0
                      |mysqld      742k  741k|mysqld       0.6|mysqld      35.0M|   0     0 |   0:   0
kjournald     0  4096B|mysqld     6032k    0 |mysqld       6.1|mysqld      35.0M|   0    16 |   0:8.70
                      |mysqld     7509k    0 |mysqld       6.2|mysqld      35.0M|   0     0 |   0:   0
[root2monitor tool]#  dstat --top-bio --top-io --top-cpu --top-mem --disk-tps --disk-util --top-oom 
----most-expensive---- ----most-expensive---- -most-expensive- --most-expensive- -dsk/total- sda--sdb- --out-of-memory---
  block i/o process   |     i/o process      |  cpu process   |  memory process |reads writs|util:util|    kill score    
init [3]    241k  331k|init [3]   9088k 1611k|kjournald    0.0|php          717M|   7    43 |0.14:0.76|httpd          13k
                      |php        3248k    0 |php          6.2|php          719M|   0     3 |   0:2.20|httpd          13k
                      |php        3312k    0 |php          6.2|php          722M|   0     3 |   0:1.90|httpd          13k
                      |php        3256k    0 |php          6.2|php          724M|   0     0 |   0:   0|httpd          13k
httpd         0   196k|httpd      8880k 1675k|php          6.2|php          726M|   0    74 |2.70:24.4|httpd          13k
kjournald     0   156k|php        3288k    0 |php          6.2|php          728M|   0     2 |   0:3.80|httpd          13k
                      |php        3280k    0 |php          6.2|php          730M|   0     0 |   0:   0|httpd          13k
kjournald     0   124k|mysqld      179M   24k|php          6.2|php          733M|   0    31 |   0:11.6|httpd          13k