September 5, 2008

Performance Testing - Load Balancer SSL Stress Testing

I am in the process of stress testing a new load balancer (F5 Big-IP 9.x). One concern with the device is its ability to handle concurrent SSL connections and terminations, and what the impact is on performance and resources. The nature off SSL makes it very CPU intensive (encryption). This is basically a torture test of HTTP GETs using SSL. What I need is a program that can keep HTTP requests active at line-speed so there are always concurrent requests being served.

The goal I am looking to reach is 50k concurrent active SSL connections going through the load balancer to a web farm.

First off, I need a load generator program. If I were to use a commercial load testing tool (LoadRunner, SilkPerformer, etc), I would most likely be unable to do a test like this due to Virtual User licensing costs. So the best way for me to achieve this is with a custom homebrewed tool. So of course I reached for Python.

The load generating machines I was given have Dual 2GHz CPUs,2GB memory, running Windows XP. The real challenge is seeing how many concurrent requests I can send from a Windows box!

First I started with a Python script that launches threads to do the HTTP (SSL) requests.

The script looks like this:
#!/usr/bin/env python
# ssl_load.py - Corey Goldberg - 2008

import httplib
from threading import Thread

threads = 250
host = '192.168.1.14'
file = '/foo.html'

def main():
    for i in range(threads):
        agent = Agent()
        agent.start()
            
class Agent(Thread):
    def __init__(self):
        Thread.__init__(self)
        
    def run(self):
        while True:
            conn = httplib.HTTPSConnection(host)
            conn.request('GET', file)
            resp = conn.getresponse()

if __name__ == '__main__':
    main()
This works great for small workloads, but Windows stops me from creating threads at around 800 per process (sometimes it was flaky and crashed after 300), so that won't work. The next step was to break the load into several processes and launch threads from each process. Rather than try to wrap this into a single script, I just use a starter script to call many instances of my load test script. I used a starter script like this:
#!/usr/bin/env python

import subprocess
processes = 60
for i in range(processes):
    subprocess.Popen('python ssl_load.py') 
250 threads per process seemed about right and didn't crash my system. Using this, I was able to launch 60 processes which gave me a total of 15k threads running.

I didn't reach the 50k goal, but on that hardware I doubt anyone could. With my 15k threads per machine, I was able to use 4 load generating machines to achieve 60k concurrent SSL connections. Not bad!

6 comments:

Grig Gheorghiu said...

Cool stuff, Corey -- I'll probably steal your script at some point :-)

Grig

Unknown said...

How did the F5 hold up under that load?

Jeff McNeil said...

...how did it hold up and which model was it? There's a pretty big difference between the hardware they'll ship.

I really like the F5 boxes. One can do some pretty cool stuff once you throw in the iRule engine.

Anonymous said...

Depending on your needs you might take a look at tsung - http://tsung.erlang-projects.org/ "Tsung's main strength is its ability to simulate a huge number of simultaneous user from a single CPU". It is easy to setup a complex sequence of requests and tsung will generate very useful stats.

Corey Goldberg said...

The F5 held up well (blanking on the model). The old hardware with the Big-IP 4.x software had a limit of 5k SSL connections. I drove the 60k and it held up perfectly.. CPU barely budged.

m3tomlins said...

Corey - you are right to point out that a comprehensive load testing tool like LoadRunner is way overkill for a simple test of connections through a load balancer. I think it would be important for you to mention the scope of your test is much more focused than an end-to-end, web application performance test.