July 19, 2010

Monitoring Stats From Memcached or Membase (Python, RRDTool)

memcached_stats_rrd.py is a script for monitoring and graphing stats from Memcached or Membase.

Memcached is a high-performance, distributed memory object caching system, and Membase is the related key-value database management system. Both are open source, with packaged/commercial versions distributed by NorthScale. They both use the memcached protocol for communication, so this script will work against a vanilla memcached installation, or against a membase server/cluster.

For more info:


script source code: memcached_stats_rrd.py

This script is useful for ad-hoc monitoring or longer term trend/capacity analysis. It collects data, stores it in RRD databases, and outputs graphs and stats in the form of PNG images. You can monitor any stats that memcached/membase publishes and graph them over specified time spans. It will generate an image file for each stat, for each time span selected.

This mini monitoring system is built with:

Sample Graph/Image Output:


(bytes_read, 3 hour timespan, 60 sec collection interval)

(mem_used, 1 hour timespan, 60 sec collection interval)

It will generate images in the directory you specify. I have an Apache web server installed serving content from the output directory for easy web viewing. You can wrap the images in some HTML and create a little dashboard to watch your entire memcached/membase cluster, like this:


(4 hour timespan, 60 sec collection interval, 2 nodes, 3 stats each)

Instructions...

1) Install prerequisites

You will need the following software installed:

  • Python 2.x
  • python-memcached (memcached client for Python)
  • RRDTool (round-robin database, logging/graphing backend)

on Debian/Ubuntu:

$ sudo apt-get install -y python-memcache
$ sudo apt-get install -y rrdtool

2) Configure the script

Near the top of the script, there are a few configuration settings:

# Config Settings
NODES = ('192.168.1.3:11211', '192.168.1.4:11211')
INTERVAL = 60
STATS = (('curr_items', 'GAUGE'), ('bytes_written', 'COUNTER'))
GRAPH_MINS = (60, 180)
GRAPH_DIR = '/var/www/'

Config Setting Definitions:

  • NODES: list of memcached/membase nodes to monitor.
  • INTERVAL: collection interval in seconds. This should be the same value as you schedule the script to run.
  • STATS: list of tuples containing (stat_name, datasource_type), where "stat_name" is a memcached/membase stat, and "datasource_type" is an RRDTool data source type (DST). The most useful data source types are GAUGE and COUNTER. GAUGE is used to report a current value, and COUNTER is used for continuous incrementing counters. (see: http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html for more info on RRD data sources)
  • GRAPH_MINS: list of minutes, corresponding to time spans displayed in the output graphs. An image for each stat is generated for each value here.
  • GRAPH_DIR: directory to generate out images in.

After you have the script configured, make the script executable:

$ chmod +x memcached_stats_rrd.py

3) Schedule the script

You can add an entry to your crontab (crontab -e) so cron will run it regularly. The example here uses a 60 sec (1 min) interval:

*/1 * * * * /home/perfserver/memcached_stats_rrd.py

[These instructions are for Linux/Unix, but you can configure a similar system on Windows using Task Scheduler instead of cron. The code in memcached_stats_rrd.py works cross-platform.]

3 comments:

Anonymous said...

I followed Your instructions and installed this on CentOS 5.3. I've used yum repos to install everything (python-memcached, rrdtool, even python-rrdtool).
I have several memcached server on same machine but different port, if that means anything, but I suppose the error is on rrd side...
Can you help me with this error?

Thank you,

Here is the error:
[code]
connecting to memcached...
2010/08/20 05:25:27 127.0.0.1 curr_items 191267
Traceback (most recent call last):
File "/usr/local/src/memcached_stats_rrd.py", line 137, in ?
main()
File "/usr/local/src/memcached_stats_rrd.py", line 49, in main
rrd.update(value)
File "/usr/local/src/memcached_stats_rrd.py", line 90, in update
raise RRDException('unable to update RRD: %s' % cmd_output.rstrip())
__main__.RRDException: unable to update RRD: rrdtool: /usr/local/lib/libpng12.so.0: no version information available (required by /usr/lib/librrd.so.4)
rrdtool: /usr/local/lib/libpng12.so.0: no version information available (required by /usr/lib/libcairo.so.2)
[/code]

Igor Ganapolsky said...

Fantastic tutorial! Looking forward to using this in my production environment.

YouthOfUnity said...

Hi Corey,
I was trying to use memcached_stats_rrd.py with memcached
VERSION 1.4.5, i get exception:
(after i turn backtrace on)
int() argument must be a string or a number, not 'NoneType'
('127.0.0.1', 11211)

[' File "memcached_stats_rrd.py", line 33, in main\n mc = memcache.Client(NODES)\n', ' File "/usr/local/lib/python2.6/dist-packages/python_memcached-1.47-py2.6.egg/memcache.py", line 168, in __init__\n self.set_servers(servers)\n', ' File "/usr/local/lib/python2.6/dist-packages/python_memcached-1.47-py2.6.egg/memcache.py", line 199, in set_servers\n self.servers = [_Host(s, self.debug) for s in servers]\n', ' File "/usr/local/lib/python2.6/dist-packages/python_memcached-1.47-py2.6.egg/memcache.py", line 988, in __init__\n self.port = int(hostData.get(\'port\', 11211))\n']
2011/04/06 11:14:51 error

so i changed :
from:
NODES = ('127.0.0.1:11211')
to
NODES = ['127.0.0.1:11211']
and it works