September 11, 2009

Python - HTTP Request Profiler

Here is a script in Python that profiles an HTTP request to a web server.

to use, give it a url:

> python


0.12454 request sent
0.24967 response received
0.49155 data transferred (9589 bytes)

It uses Python's httplib to send a GET request to the URL and adds some timing instrumentation between each step of the request/response.

Python Code:

#!/usr/bin/env python
# Corey Goldberg - September 2009

import httplib
import sys
import time

if len(sys.argv) != 2:
    print 'usage:\ \n(do not include http://)'

# get host and path names from url
location = sys.argv[1]
if '/' in location:
    parts = location.split('/')
    host = parts[0]
    path = '/' + '/'.join(parts[1:])
    host = location
    path = '/'

# select most accurate timer based on platform
if sys.platform.startswith('win'):
    default_timer = time.clock
    default_timer = time.time

# profiled http request
conn = httplib.HTTPConnection(host)
start = default_timer()  
conn.request('GET', path)
request_time = default_timer()
resp = conn.getresponse()
response_time = default_timer()
size = len(
transfer_time = default_timer()

# output
print '%.5f request sent' % (request_time - start)
print '%.5f response received' % (response_time - start)
print '%.5f content transferred (%i bytes)' % ((transfer_time - start), size)


lorg said...

Short and sweet.
To make it serious, you need three features:
1. Fetch all the linked-to files as well,
2. Do several 'complete page-fetches' simultaneously, and
3. Run for 1, 10, 20, 30, ... simultaneous page-fetches, potentially graphing the results.

You do that, and it's your classic load checker :)

It looks so simple I might do that myself, just for fun...

Corey Goldberg said...

you can check out for my full-featured HTTP performance tool.

this was just a short script I needed.

If you make any additions, let me know and we can publish them.

peterbe said...

I wrote an adaptation of your script that keeps testing over and over and displays the running average. Interested?

Corey Goldberg said...


sure, I'd love to see it. wanna email it? corey at goldb dot org

Unknown said...

It's still a top result in Google search (when I searched for "python http request profiling") :)