September 11, 2009

Python - HTTP Request Profiler

Here is a script in Python that profiles an HTTP request to a web server.

to use, give it a url:


> python http_profiler.py www.goldb.org

output:


0.12454 request sent
0.24967 response received
0.49155 data transferred (9589 bytes)

It uses Python's httplib to send a GET request to the URL and adds some timing instrumentation between each step of the request/response.

Python Code:


#!/usr/bin/env python
# Corey Goldberg - September 2009


import httplib
import sys
import time



if len(sys.argv) != 2:
    print 'usage:\nhttp_profiler.py \n(do not include http://)'
    sys.exit(1)

# get host and path names from url
location = sys.argv[1]
if '/' in location:
    parts = location.split('/')
    host = parts[0]
    path = '/' + '/'.join(parts[1:])
else:
    host = location
    path = '/'

# select most accurate timer based on platform
if sys.platform.startswith('win'):
    default_timer = time.clock
else:
    default_timer = time.time

# profiled http request
conn = httplib.HTTPConnection(host)
start = default_timer()  
conn.request('GET', path)
request_time = default_timer()
resp = conn.getresponse()
response_time = default_timer()
size = len(resp.read())
conn.close()     
transfer_time = default_timer()

# output
print '%.5f request sent' % (request_time - start)
print '%.5f response received' % (response_time - start)
print '%.5f content transferred (%i bytes)' % ((transfer_time - start), size)

5 comments:

lorg said...

Short and sweet.
To make it serious, you need three features:
1. Fetch all the linked-to files as well,
2. Do several 'complete page-fetches' simultaneously, and
3. Run for 1, 10, 20, 30, ... simultaneous page-fetches, potentially graphing the results.

You do that, and it's your classic load checker :)

It looks so simple I might do that myself, just for fun...

Corey Goldberg said...

you can check out www.pylot.org for my full-featured HTTP performance tool.

this was just a short script I needed.

If you make any additions, let me know and we can publish them.

peterbe said...

I wrote an adaptation of your script that keeps testing over and over and displays the running average. Interested?

Corey Goldberg said...

peterbe,

sure, I'd love to see it. wanna email it? corey at goldb dot org

Unknown said...

It's still a top result in Google search (when I searched for "python http request profiling") :)