July 7, 2009

Web Performance Tool Evaluation - lower end proprietary tools

I am in the middle of a Performance and Load tools selection process and wanted to get some feedback here.

I currently work in a shop that uses a mix of proprietary and open source tools for web performance & load testing. The bulk of our workload and analysis is currently done using SilkPerformer. As you all probably know, there is a class of tools that is *very* expensive (including SilkPerformer). Installations and maintenance can run into 7 figures ($$$) with yearly maintenance contracts upwards of 6 figures. Since SilkPerformer is in place and we are happy with it (besides price/maintenance), there is no point in moving to a similarly priced tool. Therefore I have ruled out the class of "high end" tools from my selection:

High-end tools
------------------------
Borland/Segue - SilkPerformer
HP/Mercury - LoadRunner
IBM/Rational - IBM Rational Performance Tester
Microfocus/Compuware - QALoad
Oracle/Empirix - Oracle Load Testing For Web Applications (e-Load)

The tool I select will be used across several web applications.. pretty straight forward HTML/AJAX/JavaScript Web UI. Here is a basic list of requirements:

Requirements
------------------------
Protocols:
- HTTP
- ODBC (SQL)

Features:
- distributed load generation
- reporting/analytics
- data driven testing
- 5000+ VU

I work on a very skilled team that is *very* proficient with programming, tools, and web technologies. Adapting to a new tool or programming language is not much of an issue.

I've searched the Open Source landscape pretty good. There are some fantastic tools (OpenSTA, JMeter, Pylot) to augment our testing, but no open source load generation tool completely meets our criteria.

Open Source tools
------------------------
OpenSTA
JMeter
Pylot

Now finally to the question/point....

I am looking at a class of tools that I will call "low-end performance tools". This includes all proprietary tools that are not listed above as "high-end tools". They tend be cheaper and more limited in functionality than the big guns, but are substantially cheaper and sometimes sufficient for complex web performance testing. This is where my interest lies. I have scoured the web and came up with a list of tools to evaluate.

Low-end tools
------------------------
Microsoft - VSTS
Radview - WebLOAD
SoftLogica - WAPT
Facilita - Forecast
Zoho - QEngine
Neotys - NeoLoad

Does anyone have any feedback or experience reports using any of the "low-end" tools listed above? Are there other tools I am overlooking that I should definitely look into?

any comments/suggestions are appreciated.

14 comments:

matt Dressman said...

we have been using FunkLoad ( http://funkload.nuxeo.org/ ) with moderate success.. It is also open-source.. I am actually in the process of moving most of our testing over to your Pylot tool.

Corey Goldberg said...

thanks matt,
I'll check out funkload also.

good luck with Pylot and post to the OpenQA forum if you need any help!

Basim Baassiri said...

There is also The Grinder

Andy said...

Have you looked at the avalanche? it's not cheap, but it's not high end either. very capable.

http://www.spirent.com/Broadband/Web_Application_Servers/Avalanche.aspx

Andrew said...

Hi Corey

I looked at low end tools about 7 years ago and used WAPT. It does the job but it was very simplistic with no flow control. Things may have changed since then.

I came across Facilita about 4 years ago it is trying to be a high end product at a low end price. It allows for complex scripts to be developed and has the reporting features you would expect from a more complex tool. Its developed by a small team so you still get the chance to “talk” to the developers when problems occur.

There is also Stress tester by Reflective Solutions which claims that you don’t need to write scripts but allows for flow control etc.

Finally there is QTest by Quotium which is on my list to look at when I get time.

I have tried categorizing the current performance tools on the market into different types. Have a look if you are interested. www.1202performance.com/?page_id=43

All the best

Andrew

Matt Drew said...

Plug for my company:

Web Performance Load Tester

http://www.webperformanceinc.com/load_testing/wplt.html

Closed source, mid-range, Eclipse-based.

Corey Goldberg said...

thanks Andrew! great info.

your link is very helpful. I will post again when I get further in my evaluation.

-Corey

Rahul Verma said...

Hi Corey,

RadView's WebLoad is pretty decent tool for web performance testing. Only trick is to build and rely on an in-house javascript library of functions rather than relying on things like automatic state management in the tool.

There's a free version available which would be good for a small scale performance test (non HTTPS).

One challenge faced would be to maintain unique logins across multiple load generators. The global varibles are not reliable.

Other challenge is the JavaScript does not support synchronisation. So, if there's a situation where mutliple virtual users manipulate a single variable across multiple load generators, one might have to come up with a robust library.

From my experience I have the following suggestions for WebLoad users:
1. Use WebLoad as more of a tool to trigger tests
2. Use only the essential sub-set of the default JavaScript functions
3. Develop your own JavaScript library
4. Do not user Agenda mix feature, rather build the same logic into your script
5. Use external utility to provide unique logins to a user on request
6. Do not use a single WebLoad console to manage more than 3000 users, it might crash

and the things work pretty well.

Regards,
Rahul Verma

Corey Goldberg said...

thanks Rahul, very good info.

I will look more into Radview... even though they stopped supporting the [crippled] open source version.

-Corey

Bhaskar Sivanathan said...

I have worked on couple of projects using Radview - WebLOAD. It is pretty decent suffice performance testing needs.

Thanks,
Bhaskar

Corey Goldberg said...

I'm not liking the looks of WebLOAD. They discontinued the Open Source version, and the commercial version is ridiculously expensive ($150k+) for my needs.

bgquest said...

I would like to know if there are any performance testing tools (open source - free) for client server applications

Adam Brown said...

As we've been mentioned by Andrew I thought I'd add some more information about QTest from Quotium (http://www.quotium.com).
In terms of price QTest is mid range (less than WebLoad by the look of the prices quoted in this blog) and high end in terms of features and capability.
It's used by large consultancies to deliver projects as it has the sophistication needed in its scripting (with correlation rules), very broad and detailed monitoring small VU footprint, distributed injector model and automated & configurable reporting.
The license is model quite unique and there is a model for consultancies as well as end users.
There is a free trial and video available on the web site.

Corey Goldberg said...

Adam,
isn't Quotium just an old fork of OpenSTA with some added features?