Web Performance Tool Evaluation - lower end proprietary tools

I am in the middle of a Performance and Load tools selection process and wanted to get some feedback here.

I currently work in a shop that uses a mix of proprietary and open source tools for web performance & load testing. The bulk of our workload and analysis is currently done using SilkPerformer. As you all probably know, there is a class of tools that is *very* expensive (including SilkPerformer). Installations and maintenance can run into 7 figures ($$$) with yearly maintenance contracts upwards of 6 figures. Since SilkPerformer is in place and we are happy with it (besides price/maintenance), there is no point in moving to a similarly priced tool. Therefore I have ruled out the class of "high end" tools from my selection:

High-end tools
------------------------
Borland/Segue - SilkPerformer
HP/Mercury - LoadRunner
IBM/Rational - IBM Rational Performance Tester
Microfocus/Compuware - QALoad
Oracle/Empirix - Oracle Load Testing For Web Applications (e-Load)

The tool I select will be used across several web applications.. pretty straight forward HTML/AJAX/JavaScript Web UI. Here is a basic list of requirements:

Requirements
------------------------
Protocols:
- HTTP
- ODBC (SQL)

Features:
- distributed load generation
- reporting/analytics
- data driven testing
- 5000+ VU

I work on a very skilled team that is *very* proficient with programming, tools, and web technologies. Adapting to a new tool or programming language is not much of an issue.

I've searched the Open Source landscape pretty good. There are some fantastic tools (OpenSTA, JMeter, Pylot) to augment our testing, but no open source load generation tool completely meets our criteria.

Open Source tools
------------------------
OpenSTA
JMeter
Pylot

Now finally to the question/point....

I am looking at a class of tools that I will call "low-end performance tools". This includes all proprietary tools that are not listed above as "high-end tools". They tend be cheaper and more limited in functionality than the big guns, but are substantially cheaper and sometimes sufficient for complex web performance testing. This is where my interest lies. I have scoured the web and came up with a list of tools to evaluate.

Low-end tools
------------------------
Microsoft - VSTS
Radview - WebLOAD
SoftLogica - WAPT
Facilita - Forecast
Zoho - QEngine
Neotys - NeoLoad

Does anyone have any feedback or experience reports using any of the "low-end" tools listed above? Are there other tools I am overlooking that I should definitely look into?

any comments/suggestions are appreciated.

XML-RPC Clients In Python and Perl

I was just writing some XML-RPC code and wanted to post some simple examples of how to talk to an XML-RPC server with some simple client-side code. Here are examples in both Python and Perl.

The examples below show how to connect to an XML-RPC server and call the service's start() method.

---

a simple XML-RPC client in Python:

#!/usr/bin/env python

import xmlrpclib

host = 'http://localhost'
port = '8888'

server = xmlrpclib.Server('%s:%s' % (host, port))
response = server.start()
print response

a simple XML-RPC client in Perl (using the Frontier-RPC module):

#!/usr/bin/perl -w

use strict;
use Frontier::Client;

my $host = 'http://localhost';
my $port = '8888';

my $server = Frontier::Client->new('url' => "$host:$port");
my $response = $server->call('start');
print $response;

Google Calls for a Joint Effort to Speed Up the Internet

check out:
http://code.google.com/speed/

writeup here:
http://www.infoq.com/news/2009/06/Google-Speed-Up-the-Internet

"Google has launched a web site in an attempt to find ways and push the speed up process of the entire Internet. Google shares research data, web site speed optimization tutorials, recorded presentations on performance, links to lots of performance optimization tools, and a discussion group inviting everyone to share ideas on how to make the web faster."

Pylot is listed in the downloads section!

Mini Web Load Tester with Python and Pylot Core

Pylot is a performance tool for benchmarking web services/applications. I am working on exposing some of Pylot's internals so you can use it as a Python Module/API for generating concurrent HTTP load.

Below is a simple function that runs a mini [multi-threaded] load test against a single URL. It will return a dictionary containing runtime statistics. Results and timing information from each request is also logged to a file.

I use something like this to run performance unit tests rapidly (10-30 secs usually). I can bang on the URL for my application and quickly see how it performs and scales.

Here is a small Python script that uses Pylot as a module:

#!/usr/bin/env python

import pylot.core.engine as pylot_engine
import os
import sys
import time
        

pylot_engine.GENERATE_RESULTS = False

url = 'http://www.pylot.org'
num_agents = 5
duration = 10
runtime_stats = {}

original_stdout = sys.stdout
sys.stdout = open(os.devnull, 'w')

req = pylot_engine.Request(url)
lm = pylot_engine.LoadManager(num_agents, 0, 0, False, runtime_stats, [])
lm.add_req(req)

lm.start()
time.sleep(duration)
lm.stop()

sys.stdout = original_stdout

for agent_num, stats in runtime_stats.iteritems():
    print 'agent %i : %i reqs : avg %.3f secs' % \
        (agent_num + 1, stats.count, stats.avg_latency)

Output:

agent 1 : 46 reqs : avg 0.220 secs
agent 2 : 46 reqs : avg 0.218 secs
agent 3 : 46 reqs : avg 0.220 secs
agent 4 : 46 reqs : avg 0.221 secs
agent 5 : 46 reqs : avg 0.221 secs

Here is a slightly larger example with some more structure and features. This creates a small command line interface for running a mini load test.

Code:

#!/usr/bin/env python
# Corey Goldberg 2009

import pylot.core.engine as pylot_engine
import os
import sys
import time



def main():
    """
    Usage: >python pylot_mini_loadtest.py   
    """
    url = sys.argv[1]
    num_agents = int(sys.argv[2])
    duration = int(sys.argv[3])
    pylot_engine.GENERATE_RESULTS = False
    print '\nmini web load test \n---------------------------------'
    agent_stats = run_loadtest(url, num_agents, duration)
    throughput = sum([stat.count for stat in agent_stats.values()]) / float(duration)
    print '%.2f reqs/sec' % throughput
    for agent_num, stats in agent_stats.iteritems():
        print 'agent %i : %i reqs : avg %.3f secs' % \
            (agent_num + 1, stats.count, stats.avg_latency)
        


def run_loadtest(url, num_agents, duration):
    """
    Runs a load test and returns a dictionary of statistics from agents.
    """
    original_stdout = sys.stdout
    sys.stdout = open(os.devnull, 'w')
    
    runtime_stats = {}
    req = pylot_engine.Request(url)
    lm = pylot_engine.LoadManager(num_agents, 0, 0, False, runtime_stats, [])
    lm.add_req(req)
    
    lm.start()
    time.sleep(duration)
    lm.stop()
    
    sys.stdout = original_stdout
    
    return runtime_stats



if __name__ == '__main__':
    main()

Usage/Output:

C:\test>python pylot_mini_loadtest.py http://www.goldb.org 8 10

mini web load test
---------------------------------
19.20 reqs/sec
agent 1 : 24 reqs : avg 0.416 secs
agent 2 : 24 reqs : avg 0.418 secs
agent 3 : 24 reqs : avg 0.417 secs
agent 4 : 24 reqs : avg 0.418 secs
agent 5 : 24 reqs : avg 0.415 secs
agent 6 : 24 reqs : avg 0.419 secs
agent 7 : 24 reqs : avg 0.419 secs
agent 8 : 24 reqs : avg 0.419 secs

Of course, for more complex scenarios, you can use the full blown tool, available at: www.pylot.org/download.html

Questions? Hit me up.

Pylot - Total Downloads So Far

Here is a graph showing total downloads of Pylot since its first release:

Decent uptake so far. Keep the downloads coming!

Pylot is a web performance testing tool. It is Free Open Source Software.

Python - Redirect or Turn Off STDOUT and STDERR

Here is an easy way to temporarily turn off STDOUT or STDERR in your Python program.

First you create a class to replace STDOUT. This is just minimal class with a 'write()' method.

class NullDevice():
    def write(self, s):
        pass

Notice its 'write()' method does nothing. Therefore, when you write to the NullDevice, output goes nowhere and is dropped. All you need to do is assign sys.stdout to this class.

Here is an example of turning STDOUT off and back on:

#!/usr/bin/env python

import sys


class NullDevice():
    def write(self, s):
        pass


print "1 - this will print to STDOUT"

original_stdout = sys.stdout  # keep a reference to STDOUT

sys.stdout = NullDevice()  # redirect the real STDOUT

print "2 - this won't print"

sys.stdout = original_stdout  # turn STDOUT back on

print "3 - this will print to SDTDOUT"

You can also do the same thing with sys.stderr to turn off STDERR.

C# - Export Windows Event Logs

Here is a little C# program to export Windows Event Logs. It reads an Event Log and prints entries to STDOUT so you can pipe the output to a file or other application.

using System;
using System.Diagnostics;


class EventLogExporter
{
    static void Main(string[] args)
    {
        EventLog evtLog = new EventLog("Application");  // Event Log type
        evtLog.MachineName = ".";  // dot is local machine
        
        foreach (EventLogEntry evtEntry in evtLog.Entries)
        {
            Console.WriteLine(evtEntry.Message);
        }
        
        evtLog.Close();
    }
}

Dell Mini 10 Netbook with Linux == Graphics FAIL

If you are planning on buying a Dell Mini 10 (or Mini 12) to run Linux, read this...

I used to have the Dell Mini 9 that came with Linux (Ubuntu 8.04). As soon as I got it, I paved it and installed Ubuntu Intrepid instead. It worked like a charm. Then I decided to sell my Mini 9 and upgrade to the Mini 10. The Mini 10 is a better machine in terms of hardware, and is MUCH better in terms of screen resolution and keyboard size (best keyboard on any netbook).

So, the Mini 10 ships with Windows installed. Since I had such good luck with the Mini 9, I figured a Linux install would be a breeze. So with my shiny new Mini 10 netbook, I tried an install Ubuntu Intrepid. It worked great but no compatible graphics driver. OK, so I waited for the Ubuntu Jaunty release and then promptly installed that. Same prob.

Here is the deal: There is no Linux driver for the graphics card it uses (Intel GMA 500). So.. if you want to run Linux on it, your only choice is to run in a non-native resolution using the default driver. This totally sucks.

There appears to be a native Linux driver somewhere (Poulsbo), but it doesn't work right now and is not packaged.

I am just running Windows for now and waiting for a real native driver to be released. Shame on Intel for not providing one.