Performance Testing - Load Balancer SSL Stress Testing

I am in the process of stress testing a new load balancer (F5 Big-IP 9.x). One concern with the device is its ability to handle concurrent SSL connections and terminations, and what the impact is on performance and resources. The nature off SSL makes it very CPU intensive (encryption). This is basically a torture test of HTTP GETs using SSL. What I need is a program that can keep HTTP requests active at line-speed so there are always concurrent requests being served.

The goal I am looking to reach is 50k concurrent active SSL connections going through the load balancer to a web farm.

First off, I need a load generator program. If I were to use a commercial load testing tool (LoadRunner, SilkPerformer, etc), I would most likely be unable to do a test like this due to Virtual User licensing costs. So the best way for me to achieve this is with a custom homebrewed tool. So of course I reached for Python.

The load generating machines I was given have Dual 2GHz CPUs,2GB memory, running Windows XP. The real challenge is seeing how many concurrent requests I can send from a Windows box!

First I started with a Python script that launches threads to do the HTTP (SSL) requests.

The script looks like this:

#!/usr/bin/env python
# ssl_load.py - Corey Goldberg - 2008

import httplib
from threading import Thread

threads = 250
host = '192.168.1.14'
file = '/foo.html'

def main():
    for i in range(threads):
        agent = Agent()
        agent.start()
            
class Agent(Thread):
    def __init__(self):
        Thread.__init__(self)
        
    def run(self):
        while True:
            conn = httplib.HTTPSConnection(host)
            conn.request('GET', file)
            resp = conn.getresponse()

if __name__ == '__main__':
    main()

This works great for small workloads, but Windows stops me from creating threads at around 800 per process (sometimes it was flaky and crashed after 300), so that won't work. The next step was to break the load into several processes and launch threads from each process. Rather than try to wrap this into a single script, I just use a starter script to call many instances of my load test script. I used a starter script like this:

#!/usr/bin/env python

import subprocess
processes = 60
for i in range(processes):
    subprocess.Popen('python ssl_load.py') 

250 threads per process seemed about right and didn't crash my system. Using this, I was able to launch 60 processes which gave me a total of 15k threads running.

I didn't reach the 50k goal, but on that hardware I doubt anyone could. With my 15k threads per machine, I was able to use 4 load generating machines to achieve 60k concurrent SSL connections. Not bad!