Pylot - Web Load Testing from Amazon Elastic Compute Cloud (EC2)

I have been playing around with Amazon's Elastic Compute Cloud (EC2). It allows you to provision virtual machines on-demand and configure and control your own compute clusters. To do external (over a WAN) performance\load testing against your web application or services, you can put together a cloud based test harnesses using some simple tools. My open source tool Pylot does the job well for simple web performance and load tests. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI or shell/console. You define your test cases in an XML file. This is where you specify the requests (url, method, body/payload, etc) and verifications.

It is incredibly easy to provision an instance and launch a virtual machine using the EC2 console. Once you have it up and running, you can just connect to your new virtual machine (via RDP or SSH) and get started.

Here is a screenshot of my terminal session using Pylot (click to enlarge):

You can see I took the following steps:

1. remotely login to the EC2 instance using SSH and my private key
2. download Pylot using wget
3. unzip the distribution
4. change to the Pylot directory
5. launch the default test with 1 agent (virtual user)

Thats it! I ran a test today doing 1500 Virtual Users from one instance of 64-bit Fedora Linux, and it took me about 5 minutes to get it setup and running. Python 2.5 is already installed on the image, and no further configuration is needed to run a basic test. To run the Pylot GUI and generate results graphs, you need to install wxPython and Matplotlib.

You can upload your own test case files and run any load test scenario you want. Don't forget, you can create and save your own machine images and launch as many as you want to run a large distributed test.

I'd like to build some tools to make this easier and to create preconfigured machine images for other people to start with. It needs common results collection and remote control of user agents. I'm thinking of a cluster of virtual machines all running Pylot, each running thousands of virtual user agents. EC2 provides the ability to place instances in multiple locations, so this could be used to create a massive geo-distributed test bed with capacity on-demand.

More soon...

Pylot Version 1.20 Released - Open Source Web Performance Tool

The 1.20 release of Pylot is out!

Go grab a copy at: http://www.pylot.org/download.html

"Pylot is a free open source tool for testing performance and scalability of web services. It runs HTTP load tests, which are useful for capacity planning, benchmarking, analysis, and system tuning. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI or shell/console."

new features include:

• refactored transaction engine with lower memory footprint and disk i/o
• automatic cookie handling
• better results reports
• test naming
• specify output location
• specify test case file
• bug fixes

To get started, visit Getting Started Guide: http://www.pylot.org/gettingstarted.html

Post your questions and feedback in the Pylot forum: http://clearspace.openqa.org/community/pylot

Mark Rogers and I had a nice little hackathon getting this release put together. Special thanks to Mark and the other patch submitters for helping out.

Screenshots:

Python - Send Email From Windows Using CDO

Here is quick script showing how to use Python to send email from Windows.

This approach uses the Python For Windows Extensions to access Outlook/Exchange with CDO (Collaboration Data Objects).

#!/usr/bin/env python
# Corey Goldberg

from win32com.client import Dispatch

session = Dispatch('MAPI.session')
session.Logon('','',0,1,0,0,'exchange.foo.com\nUserName');
msg = session.Outbox.Messages.Add('Hello', 'This is a test')
msg.Recipients.Add('Corey', 'SMTP:corey@foo.com')
msg.Send()
session.Logoff()

Pylot - Help/Discussion Forum Launched

Thanks to OpenQA, I launched a web forum for help and discussions about Pylot:
Pylot Forums

This forum is a place for Pylot users and developers to share: questions, comments, experiences, enhancement requests, bug reports, etc. If you need any help, this is the place to post

---

What is Pylot?

"Pylot is a free open source tool for testing performance and scalability of web services. It runs HTTP load tests, which are useful for capacity planning, benchmarking, analysis, and system tuning.

Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI or shell/console."

The official Pylot website is located at: www.pylot.org

Pylot Testimonial - Easy Setup for Web Load Testing

(Pylot is a an open source web performance/load testing tool I developed)

I just saw this post from Mark (f4nt) about load/performance testing Confluence (enterprise Wiki software). He briefly talks about performance tool selection and why he chose Pylot as his tool.

Load Testing Confluence

"A small part of my problem of getting my tests rolling was finding a test suite that suited my needs. There’s a ton of potential options such Grinder, Bench, Browsermob, WebInject, HttPerf, funkload, and so on and so forth. I do want to use BrowserMob, just because it’s incredibly slick, looks easy to use, and looks to be quite reliable. Unfortunately, it’s not free though, and my budget for this testing is currently $0 :). My big problem with a few of the load testing applications is that they were a major pain just to setup, and get rolling. JMeter kept crashing, and was more effort to setup than I wanted to deal with. Grinder, again, didn’t seem to have an easy setup method. Granted, I could have made either of these work, and it wouldn’t have killed me. The fact of the matter was though that I just don’t have time to deal with these items. While I am doing these tests for work, I’m mainly doing them off hours because that’s when I have time to actually do it. Hence, I wanted something that could give me basic statistics, that I could setup quickly, that was free. Considering my slant towards open source software and python, the application I would use being written in Python and being GPLed were bonuses.

That brings us to Pylot. It’s free, it’s released under the GPL, and it’s written in Python. As a bonus, the tests are simple to setup, the result output is usable, and making modifications to the application was easy as well. This allows me to quickly create test scenarios and pound away at the application I’m testing with little to no fuss whatsoever. Creating tests is just a matter of modifying a simple XML to place the URLs you wish to hit. You can have post requests as well without any major trouble. Whatever it is you want to do, seems to be quite plausible in the grand scheme of things. Then, when you run into something you can’t do, modifying the code itself to make it do what you want isn’t hard at all either."

Thanks for using and giving props to Pylot, Mark!

Python - Read Outlook Email and Save Embedded Attachments

I was struggling with this for days, so I figured I should post the code so others can see how to do it.

The following script will read the last email in an Outlook mailbox and save the attachments. It uses the CDO COM Interface to interact with Outlook/Exchange.

#!/usr/bin/env python
# Read the last email in an Outlook mailbox and save the attachments.

from win32com.client import Dispatch


def main():   
    session = Dispatch('MAPI.session')
    #session.Logon('Outlook')  # for local mailbox
    session.Logon('','',0,1,0,0,'exchange.foo.com\nusername');
    inbox = session.Inbox
    message = inbox.Messages.GetLast()
    attachments = message.Attachments
    for i in range(attachments.Count):
        attachment = attachments.Item(i + 1) # indexes are 1 based
        filename = 'c:\\tempfile_%i' % i
        attachment.WriteToFile(filename)
    session.Logoff()


if __name__ == '__main__':
    main()

* Extra Special thanks to Sergey Golovchenko for helping out.

Capacity Planning - A Recession is All About Doing More With Less

I liked this quote a lot:

"A recession is all about doing more with less or, at least, with the capital equipment already purchased. That's a major reason for doing capacity planning."
(from Guerrilla Training Schedule 2009)

I never thought of a recession in those terms. But perhaps this is a boon for Performance Engineers into testing/tuning/scaling and capacity planning. [shrug] Who knows.

Weird Python 3.0 / SciTE Editor Issue - Console Output

My editor (SciTE) has always worked fine for programming Python 2.x. I'm now trying some Python 3.0 code and ran into an issue on my first script. The issue doesn't happen running the same code under 2.x. It is really confusing and annoying me and I have no idea who to file a bug with.

For whatever reason, as soon as I call an http.client request() method from within a while loop, nothing further is printed to the editor's console (stdout). If it is not in a loop, I get the output. The script executes fine aside from printing output. If I run the script from a regular command prompt (outside of SciTE), it works fine also.

Setup:
* Windows (tested on XP and Vista)
* Python 3.0 Final
* SciTE Version 1.75-wbd-1

This works: ('foo' is printed once to console)

import http.client

conn = http.client.HTTPConnection('www.goldb.org')
conn.request('GET', '/')
conn.close()
print('foo')

This works: ('foo' is printed repeatedly to console)

import http.client

while True:
    conn = http.client.HTTPConnection('www.goldb.org')
    # conn.request('GET', '/')
    conn.close()
    print('foo')

This doesn't work: (nothing is printed to console)

import http.client

while True:
    conn = http.client.HTTPConnection('www.goldb.org')
    conn.request('GET', '/')
    conn.close()
    print('foo')

This same exact setup works fine in Python 2.x. I've also tried starting Python with the '-u' option to get unbuffered output.

Anyone have ANY clue what could be going on?

Update: This seems to be related to Issue 4705 at bugs.python.org. It has to do with how Python 3.0 does (or doesn't) do unbuffered I/O. A patch was already submitted. Hope it's fixed in next release.