Log book: 2月 2017

2017年2月18日星期六

Install Cacti 1.0, spine in Ubuntu 16 with MySQL 5.7

NOTE:
As of Feb 18, 2017, Ubuntu is still packaged with Cacti 0.8.8f and spine 0.8.8b, so I will show here how to install the latest Cacti 1.0.x manually. Well, it is not that 0.8.8 is no good anymore, but many of it's useful plugins are quite broken under php 7 and MySQL 5.7... So if you really need to stick with 0.8.8 version, I would suggest not to do that in Ubuntu 16; otherwise, just install the latest 1.0.x.

Cacti official setup documentation: http://docs.cacti.net/manual:100:1_installation#requirements

Packages required:
gcc
apache2, libapache2-mod-php
snmpd, snmp, snmp-mibs-downloader, libsnmp, libsnmp-dev
libssl-dev
libc-dev
libc6-dev
php, php-snmp, php-xml, php-gd, php-common, php-gmp, php-ldap, php-mbstring, php-mysql,
mysql-server, mysql-client, libmysqlclient-dev
help2man
rrdtools
git

NOTE: make sure your system timezone and php.ini timezone is setup correctly (I have carelessly set a wrong system timezone and took me a lot of time to find out my data in rrd file is set in the wrong timeslot...)
In my case, type the following command "timedatectl set-timezone Asia/Hong_Kong"
vi /etc/php/7.0/apache2/php.ini
=> date.timezone = Asia/Hong_Kong

Download the sources
Visit Cacti official home and check on latest versions and patches: http://www.cacti.net/

wget http://www.cacti.net/downloads/cacti-1.0.3.tar.gz
tar xvzf cacti-1.0.3.tar.gz
mv cacti-1.0.3 /var/www/html/cacti
## make sure apache have full access to the folder
chown -R www-data:www-data /var/www/html/cacti

vi /etc/apache2/conf-enabled/cacti.conf

Alias /cacti /var/www/html/cacti
<Directory /var/www/html/cacti>
Options +FollowSymLinks
AllowOverride None
<IfVersion >= 2.3>
Require all granted
</IfVersion>
<IfVersion < 2.3>
Order Allow,Deny
Allow from all
</IfVersion>

AddType application/x-httpd-php .php

<IfModule mod_php.c>
php_flag magic_quotes_gpc Off
php_flag short_open_tag On
php_flag register_globals Off
php_flag register_argc_argv On
php_flag track_vars On
# this setting is necessary for some locales
php_value mbstring.func_overload 0
php_value include_path .
</IfModule>

DirectoryIndex index.php
</Directory>

## Configure database and install cacti
# set root password
mysqladmin password

cd /var/www/html/cacti
vi include/config.php ## change username/password as you like, which you would be using for database privileges setup
mysql -p
> CREATE DATABASE cacti
> grant all privileges on cacti.* to cactiuser@'localhost' identified by 'cactiuser';
> grant select on mysql.time_zone_name to cactiuser@'localhost' identified by 'cactiuser';

## tuning MySQL configurations
## this is just an example, you may need much larger values if you have a large site

## heap table size is specifically for performance enhancement, please refer to the following URL for details

http://logch.blogspot.hk/2017/02/tuning-cacti-10-performance-for-medium.html

/etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]

collation-server = utf8_general_ci

character-set-server = utf8

max_heap_table_size = 256M

max_allowed_packet = 16777216

tmp_table_size = 64M

join_buffer_size = 64M

innodb_file_per_table = on

innodb_doublewrite = off

innodb_additional_mem_pool_size = 80M

innodb_flush_log_at_trx_commit = 2

## create tables
mysql -p cacti < cacti.sql
## populate mysql time_zone_name table
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -p mysql

## now open a browser and open "http://your_server_ip/cacti/" and just follow the instructions. If it says you are missing some modules, just apt install them.
https://www.youtube.com/watch?v=rqK5OnbF1BY (although this video is done under CentOS 7, but the web setup process is mostly the same)

## Compile spine

wget http://www.cacti.net/downloads/spine/cacti-spine-1.0.3.tar.gz
tar xvzf cacti-spine-1.0.3.tar.gz
cd cacti-spine-1.0.3
./configure
make
make install

## Download thold plugin
# chdir to cacti plugins folder
cd /var/www/html/cacti/plugins
# download the source
git clone -b master https://github.com/Cacti/plugin_thold.git
# rename it to thold, yes it matters...

mv plugin_thold thold

Install Thold from the UI and a simple example can be found in video below (start at around 9:15)

https://www.youtube.com/watch?v=rqK5OnbF1BY

Upgrade from Cacti 1.0.3 to Cacti 1.0.x

# cd /var/www/html

# wget http://www.cacti.net/downloads/cacti-1.0.x.tar.gz

# tar xvzf cacti-1.0.3.tar.gz

# mv cacti cacti-1.0.3-old

# mv cacti-1.0.x cacti

Update cacti/include/config.php on the database username and password

Open a web browser and open cacti url, e.g. http://x.x.x.x/cacti/, follow the steps and choose upgrade.

# cp -p cacti-1.0.3-old/rra/* cacti/rra/

Also copy whatever you have added by yourself from cacti-1.0.3-old/scripts/* and cacti-1.0.3-old/resource/* to cacti/scripts and cacti/resource. That's it.

2017年2月12日星期日

Tuning Cacti 1.0.x performance for large sites, running on CentOS 7 with MariaDB

Menu => Settings => Poller
Poller Type => spine

Menu => Settings => Paths
Spine Binary File Location => /usr/local/spine/bin/spine
Spine Config File Path => /usr/local/spine/etc/spine.conf

We don't have a config file yet, so
# cp /usr/local/spine/etc/spine.conf.dist /usr/local/spine/etc/spine.conf
# vi /usr/local/spine/etc/spine.conf
... modify DB_User, DB_Pass etc

Menu => System Utilities => View Boost Status

By default, cacti 1.0.1 has chosen InnoDB engine for Boost Storage, and that is why it shows unlimited on table size and maximum records (well, unlimited as long as your hard disk has enough free space). As I remember, the DB engine used to be "MEMORY" in Cacti 0.8.x with the "boost" plugin, which would give maximum performance with some limitations. First, the maximum allowed table size and records would have a limit, i.e. the amount of RAM, and you really don't want to go over that limit. Second, you will have to do some calculations in order to have the configuration set correctly.
...

Boost Storage Statistics
Database Engine:	InnoDB
Current Boost Table(s) Size:	16.00 KBytes
Avg Bytes/Record:	2 KBytes
Max Allowed Boost Table Size:	Unlimited
Estimated Maximum Records:	Unlimited Records

...

Steps:

Goto Menu => Settings => Performance
My initial settings are
- Enable On-demand RRD Updating => Checked
- How Often Should Boost Update All RRDs => 1 hour
- Maximum Records => 100000
- Enable direct population of poller_output_boost table => Checked
- Enable Image Caching => Checked
- Location for Image Files => /var/www/html/cacti/cache/boost/

Configure database
Configure heap table size, and restart MariaDB. "128M" is just an example, it is likely to be on gigabyte level for large sites.

#vi /etc/my.cnf.d/server.cnf
[mysqld]
max_heap_table_size = 128M

systemctl restart mariadb

# mysql -p cacti

> alter table poller_output_boost ENGINE=MEMORY;

Now go back to Menu => System Utilities => View Boost Status

The engine type has become "MEMORY" and the table size is no longer unlimited.

Boost Storage Statistics
Database Engine:	MEMORY
Current Boost Table(s) Size:	124.28 KBytes
Avg Bytes/Record:	25 KBytes
Max Record Length:	30 Bytes
Max Allowed Boost Table Size:	120.35 MBytes
Estimated Maximum Records:	4.958 K Records

If you do some math, you may notice that the data is taking more storage than it should. This is because in MySQL/MariaDB, MEMORY tables use fixed-length row-storage format. So the default varchar(512) in the output column would just become char(512) and thus waste a lot of space. If you know your polling results would not exceed a certain length, please do adjust the varchar length for the output column.

e.g. alter table poller_output_boost modify column output varchar(128);

So we have "Maximum records" set to 100,000 and update interval set to hourly. Now you have to calculate whether your heap table can hold 100000 records or not. You see the "Estimated Maximum Records" is only 4.957K here, and say "oh no it's gonna blow up my heap table". Well this is not exactly true. The "Avg bytes/record" is 25K bytes, which is not possible if you look at the poller_output_boost table structure seriously. This is only because database system is allocating memory in chunks, in this case ~128K a time.

From the "show table..." statement below, we know that the maximum data length is 109506040, and the average row length is 452. So we have 109506040 / 452 bytes => 200,000+ records

> desc poller_output_boost;

+---------------+-----------------------+------+-----+---------------------+-------+

+---------------+-----------------------+------+-----+---------------------+-------+

| local_data_id | mediumint(8) unsigned | NO | PRI | 0 | |

| rrd_name | varchar(19) | NO | PRI | | |

| time | timestamp | NO | PRI | 0000-00-00 00:00:00 | |

+---------------+-----------------------+------+-----+---------------------+-------+

> show table status where name ='poller_output_boost';

+---------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+

| poller_output_boost | MEMORY | 10 | Fixed | 35 | 452 | 127224 | 109506040 | 1596 | 0 | NULL | 2017-02-12 23:50:52 | NULL | NULL | utf8_unicode_ci | NULL | | |

Finally, since the intermediate polling results are in memory, please remember to flush the data when you need to restart the database or the server.

# cd /var/www/html/cacti

# php poller_boost.php --force

2017年2月6日星期一

Django, uwsgi, nginx setup in Ubuntu 16

Video URL: https://www.youtube.com/watch?v=TYZfHn0MoXg

Detailed procedures:

apt install python
apt install python-pip
apt install nginx
pip install virtualenv
pip install virtualenvwrapper
pip install uwsgi

adduser deploy
su - deploy
## for virtualenv projects
mkdir projects
## for django webpages
mkdir web

## add virtualenv environment variables to user deploy
vi /home/deploy/.bashrc
...
export WORKON_HOME=~/projects
source /usr/local/bin/virtualenvwrapper.sh

mkvirtualenv p1

pip install django

cd ~/web/

django-admin.py startproject d1

cd d1

./manage.py migrate

vi d1/settings.py

ALLOWED_HOSTS = ['172.25.0.103']

...

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

./manage.py collectstatic

./manage.py runserver 0.0.0.0:5000

##open web brower to test

## quit virtualenv

deactivate

## change back to root or sudo -i

## test uwsgi with django

uwsgi --master --http :5000 --home /home/deploy/projects/p1 --chdir /home/deploy/web/d1 --module d1.wsgi:application

## open web brower to test, and you will find the images and styles are gone, as uwsgi doesn't know where to service the static content from django. We will get to that in nginx configuration part.

## create a service to start uwsgi automatically

vi /etc/systemd/system/uwsgi.service

[Unit]

Description=uWSGI Emperor

[Service]

ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown deploy:www-data /run/uwsgi'

ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites

Restart=always

KillSignal=SIGQUIT

Type=notify

StandardError=syslog

NotifyAccess=all

[Install]

WantedBy=multi-user.target

## create config file for uwsgi

vi /etc/uwsgi/sites/p1.ini

[uwsgi]

uid = deploy

base = /home/%(uid)

chdir = %(base)/web/d1

home = %(base)/projects/p1

module = d1.wsgi:application

master = true

processes = 5

socket = /run/uwsgi/p1.sock

chown-socket = %(uid):www-data

chmod-socket = 660

vacuum = true

## start uwsgi service

systemctl restart uwsgi.service

## create nginx config file

/etc/nginx/sites-available/p1

server {

listen 9090;

server_name 172.25.0.103;

location = /favicon.ico { access_log off; log_not_found off; }

location /static/ {

root /home/deploy/web/d1;

}

location / {

include uwsgi_params;

uwsgi_pass unix:/run/uwsgi/p1.sock;

}

## sym-link to sites-enable

ln -s /etc/nginx/sites-available/p1 /etc/nginx/sites-enable/

systemctl restart nginx

2017年2月5日星期日

Install cacti 1.x, spine in CentOS 7 with mariadb (right, not mysql)

UPDATE:
Since new release is out, I added the procedures for upgrading cacti 1.0.1 to 1.0.x; and from 1.0.x to 1.1.x

NOTE:
Wouldn't it be better to use Ubuntu as cacti is packaged up and is so easy to install. Actually yes and no. I have tried that on Ubuntu 16, and yes, it is just a few "apt-get install" and we are good to go. But there are other problems as of Feb 2017, cacti 0.8.8h (from apt-get) is not very ready for php 7 and MySQL 5.7, especially on the plugins contributed from the community all along the years. For instance, "split" function is gone in php 7 and it is quite common in the Cacti code base. Another useful plugin "thold" for sending out alarms has even more serious problems with MySQL 5.7 as the SQL checking have become much more "strict" (I have tried turning off strict mode but still no luck). But to be fair, it is not MySQL's fault really, "thold" is written in a rather "relax" way, and older MySQL would just let it pass with no problem.
So instead going through all the troubles, why not install the latest 1.x, though there is no yum or apt-get... But it is really nice that the new Cacti 1.0 has absorbed many useful plugins into the core code base, such as boost, autom8, ugroup etc. I am not sure why thold is left out as it provides the basic alarming functionality that is what Cacti lacks.
After all, I think it would be still interesting to see if Cacti 1.0 can run smoothly in Ubuntu 16 as php 7 is supposed to have a much higher performance than php 5

Cacti official setup documentation: http://docs.cacti.net/manual:100:1_installation#requirements

Pre-req:
yum install net-snmp
yum install netsnmp-devel
yum install net-snmp-utils
yum install mariadb
yum install mariadb-devel
yum install gcc
yum install help2man
yum install rrdtools
yum install php
yum install php-snmp
yum install php-process
yum install php-mysql
yum install php-pdo
yum install php-ldap
yum install php-mbstring
yum install php-xml
yum install git

## Configure database and install cacti
# set root password
mysqladmin password

cd /var/www/html/cacti
vi include/config.php ## change username/password as you like, which you would be using for database privileges setup
mysql -p
> CREATE DATABASE cacti
> grant all privileges on cacti.* to cactiuser@'localhost' identified by 'cactiuser';
> grant select on mysql.time_zone_name to cactiuser@'localhost' identified by 'cactiuser';

## tuning Maria DB configurations (the syntax is just like mysql, so it is quite easy)
/etc/my.cnf.d/client.cnf
[client]
default-character-set=utf8

## this is just an example, you may need much larger values if you have a large site

## heap table size is specifically for performance enhancement, please refer to the following URL for details

http://logch.blogspot.hk/2017/02/tuning-cacti-10-performance-for-medium.html

/etc/my.cnf.d/server.cnf [mysqld]

collation-server = utf8_general_ci

init-connect='SET NAMES utf8'

character-set-server = utf8

max_heap_table_size = 256M

max_allowed_packet = 16777216

tmp_table_size = 64M

join_buffer_size = 64M

innodb_file_per_table = on

innodb_doublewrite = off

innodb_additional_mem_pool_size = 80M

innodb_flush_log_at_trx_commit = 2

## create tables
mysql -p cacti < cacti.sql
## populate mysql time_zone_name table
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -p mysql

## now open a browser and open "http://your_server_ip/cacti/" and just follow the instructions. If it says you are missing some modules, just yum install them.
https://www.youtube.com/watch?v=rqK5OnbF1BY

## Compile spine

wget http://www.cacti.net/downloads/spine/cacti-spine-1.0.1.tar.gz
tar xvzf cacti-spine-1.0.1.tar.gz
cd cacti-spine-1.0.1
./configure
make
make install

## Use spine
cd /usr/local/spine/etc/
(1) copy the "spine.conf.dist" to "spine.conf", and modify the DB related parameters inside, e.g. db name, user/pass (2) Then in Cacti UI -> Settings -> Paths, input the spine directories, e.g. Binary: /usr/local/spine/bin/spine Config: /usr/local/spine/etc/spine.conf (3) Settings -> Poller, choose spine instead of cmd.php

## Download thold plugin
# chdir to cacti plugins folder
cd /var/www/html/cacti/plugins
# download the source
git clone -b master https://github.com/Cacti/plugin_thold.git
# rename it to thold, yes it matters...

mv plugin_thold thold

Install Thold from the UI and a simple example can be found in video below (start at around 9:15)
https://www.youtube.com/watch?v=rqK5OnbF1BY

Upgrade from Cacti 1.0.1 to Cacti 1.0.x
# cd /var/www/html
# wget http://www.cacti.net/downloads/cacti-1.0.3.tar.gz
# tar xvzf cacti-1.0.3.tar.gz
# mv cacti cacti-1.0.1-old
# mv cacti-1.0.3 cacti
Update cacti/include/config.php on the database username and password
Open a web browser and open cacti url, e.g. http://x.x.x.x/cacti/, follow the steps and choose upgrade.

# cp -p cacti-1.0.1-old/rra/* cacti/rra/
Also copy whatever you have added by yourself from cacti-1.0.1-old/scripts/* and cacti-1.0.1-old/resource/* to cacti/scripts and cacti/resource. That's it.

Upgrade from Cacti 1.0.x to Cacti 1.1.x
It should be pretty much the same
# cd /var/www/html
# wget http://www.cacti.net/downloads/cacti-1.0.3.tar.gz
# tar xvzf cacti-1.1.2.tar.gz
# mv cacti cacti-1.0.3-old
# mv cacti-1.1.2 cacti
Update cacti/include/config.php on the database username and password
Open a web browser and open cacti url, e.g. http://x.x.x.x/cacti/, follow the steps and choose upgrade.

Sample screen you would see in browser

訂閱：文章 (Atom)

2017年2月18日 星期六

Install Cacti 1.0, spine in Ubuntu 16 with MySQL 5.7

2017年2月12日 星期日

Tuning Cacti 1.0.x performance for large sites, running on CentOS 7 with MariaDB

2017年2月6日 星期一

Django, uwsgi, nginx setup in Ubuntu 16

2017年2月5日 星期日

Install cacti 1.x, spine in CentOS 7 with mariadb (right, not mysql)

2017年2月18日星期六

2017年2月12日星期日

2017年2月6日星期一

2017年2月5日星期日