I am in the middle of a Performance and Load tools selection process and wanted to get some feedback here.
I currently work in a shop that uses a mix of proprietary and open source tools for web performance & load testing. The bulk of our workload and analysis is currently done using SilkPerformer. As you all probably know, there is a class of tools that is *very* expensive (including SilkPerformer). Installations and maintenance can run into 7 figures ($$$) with yearly maintenance contracts upwards of 6 figures. Since SilkPerformer is in place and we are happy with it (besides price/maintenance), there is no point in moving to a similarly priced tool. Therefore I have ruled out the class of "high end" tools from my selection:
High-end tools ------------------------ Borland/Segue - SilkPerformer HP/Mercury - LoadRunner IBM/Rational - IBM Rational Performance Tester Microfocus/Compuware - QALoad Oracle/Empirix - Oracle Load Testing For Web Applications (e-Load)
The tool I select will be used across several web applications.. pretty straight forward HTML/AJAX/JavaScript Web UI. Here is a basic list of requirements:
Requirements ------------------------ Protocols: - HTTP - ODBC (SQL) Features: - distributed load generation - reporting/analytics - data driven testing - 5000+ VU
I work on a very skilled team that is *very* proficient with programming, tools, and web technologies. Adapting to a new tool or programming language is not much of an issue.
I've searched the Open Source landscape pretty good. There are some fantastic tools (OpenSTA, JMeter, Pylot) to augment our testing, but no open source load generation tool completely meets our criteria.
Open Source tools ------------------------ OpenSTA JMeter Pylot
Now finally to the question/point....
I am looking at a class of tools that I will call "low-end performance tools". This includes all proprietary tools that are not listed above as "high-end tools". They tend be cheaper and more limited in functionality than the big guns, but are substantially cheaper and sometimes sufficient for complex web performance testing. This is where my interest lies. I have scoured the web and came up with a list of tools to evaluate.
Low-end tools ------------------------ Microsoft - VSTS Radview - WebLOAD SoftLogica - WAPT Facilita - Forecast Zoho - QEngine Neotys - NeoLoad
Does anyone have any feedback or experience reports using any of the "low-end" tools listed above? Are there other tools I am overlooking that I should definitely look into?
any comments/suggestions are appreciated.
14 comments:
we have been using FunkLoad ( http://funkload.nuxeo.org/ ) with moderate success.. It is also open-source.. I am actually in the process of moving most of our testing over to your Pylot tool.
thanks matt,
I'll check out funkload also.
good luck with Pylot and post to the OpenQA forum if you need any help!
There is also The Grinder
Have you looked at the avalanche? it's not cheap, but it's not high end either. very capable.
http://www.spirent.com/Broadband/Web_Application_Servers/Avalanche.aspx
Hi Corey
I looked at low end tools about 7 years ago and used WAPT. It does the job but it was very simplistic with no flow control. Things may have changed since then.
I came across Facilita about 4 years ago it is trying to be a high end product at a low end price. It allows for complex scripts to be developed and has the reporting features you would expect from a more complex tool. Its developed by a small team so you still get the chance to “talk” to the developers when problems occur.
There is also Stress tester by Reflective Solutions which claims that you don’t need to write scripts but allows for flow control etc.
Finally there is QTest by Quotium which is on my list to look at when I get time.
I have tried categorizing the current performance tools on the market into different types. Have a look if you are interested. www.1202performance.com/?page_id=43
All the best
Andrew
Plug for my company:
Web Performance Load Tester
http://www.webperformanceinc.com/load_testing/wplt.html
Closed source, mid-range, Eclipse-based.
thanks Andrew! great info.
your link is very helpful. I will post again when I get further in my evaluation.
-Corey
Hi Corey,
RadView's WebLoad is pretty decent tool for web performance testing. Only trick is to build and rely on an in-house javascript library of functions rather than relying on things like automatic state management in the tool.
There's a free version available which would be good for a small scale performance test (non HTTPS).
One challenge faced would be to maintain unique logins across multiple load generators. The global varibles are not reliable.
Other challenge is the JavaScript does not support synchronisation. So, if there's a situation where mutliple virtual users manipulate a single variable across multiple load generators, one might have to come up with a robust library.
From my experience I have the following suggestions for WebLoad users:
1. Use WebLoad as more of a tool to trigger tests
2. Use only the essential sub-set of the default JavaScript functions
3. Develop your own JavaScript library
4. Do not user Agenda mix feature, rather build the same logic into your script
5. Use external utility to provide unique logins to a user on request
6. Do not use a single WebLoad console to manage more than 3000 users, it might crash
and the things work pretty well.
Regards,
Rahul Verma
thanks Rahul, very good info.
I will look more into Radview... even though they stopped supporting the [crippled] open source version.
-Corey
I have worked on couple of projects using Radview - WebLOAD. It is pretty decent suffice performance testing needs.
Thanks,
Bhaskar
I'm not liking the looks of WebLOAD. They discontinued the Open Source version, and the commercial version is ridiculously expensive ($150k+) for my needs.
I would like to know if there are any performance testing tools (open source - free) for client server applications
As we've been mentioned by Andrew I thought I'd add some more information about QTest from Quotium (http://www.quotium.com).
In terms of price QTest is mid range (less than WebLoad by the look of the prices quoted in this blog) and high end in terms of features and capability.
It's used by large consultancies to deliver projects as it has the sophistication needed in its scripting (with correlation rules), very broad and detailed monitoring small VU footprint, distributed injector model and automated & configurable reporting.
The license is model quite unique and there is a model for consultancies as well as end users.
There is a free trial and video available on the web site.
Adam,
isn't Quotium just an old fork of OpenSTA with some added features?
Post a Comment