December 14, 2009

http_rrd_profiler - HTTP Request Profiler with RRD Storage and Graphing

http_rrd_profiler is an HTTP testing/monitoring tool written in Python. It uses Python's httplib to send a GET request to a web resource and adds performance timing instrumentation between each step of the request/response. RRDTool is used for data storage and graph generation. RRDtool is an open source, high performance logging and graphing system for time-series data.

The RRD graphs it produces:



The code is located in my repository. You will need the following files:
http_rrd_profiler.py
rrd.py


Usage:

http_rrd_profiler.py <host> <interval in secs>

You invoke it from the command like like this:

> python http_rrd_profiler.py www.goldb.org 5

This will send an HTTP GET to http://www.goldb.org/ every 5 seconds.

Timing results are printed to the console and stored in a Round Robin Database (RRD). During each interval, a graph (png image) is generated.

Console output:

----------------
216 request sent
337 response received
456 content transferred (8044 bytes)
----------------
208 request sent
332 response received
453 content transferred (8044 bytes)
----------------
212 request sent
332 response received
452 content transferred (8044 bytes)
----------------
210 request sent
329 response received
448 content transferred (8044 bytes)
----------------
214 request sent
333 response received
452 content transferred (8044 bytes)

* tested on Ubuntu Karmic and WinXP with Python 2.6

3 comments:

Skyler Call said...

No matter how long I tell it to wait between checks it always takes twice as long as it should. For example, when I tell it to check every 5 seconds it checks every 10 and when I tell it to check every 120 seconds it checks every 240. I am running this on CentOS in VMWare Server. Also, after checking 11 times it spit out this error and quit:

Traceback (most recent call last):
File "http_rrd_profiler.py", line 78, in ?
main()
File "http_rrd_profiler.py", line 46, in main
time.sleep(interval - elapsed)
IOError: [Errno 22] Invalid argument

Corey Goldberg said...

skyler, the timing seems to work fine for me (tested on WinXP and Ubuntu Karmic, both with Python 2.6).

Unknown said...

It's Vmware Fault! Not every processor cycle is devoted to your VM. Every other cycle is given to the host system.

See if you can up the priority to that VM.