Recently I was looking at the scalability of a web platform, and had to perform some stress analysis on it to evaluate whether or not it would accommodate the potential user base for a new application.

The idea was to pose the question:

"N percentage of web pages should load in X seconds, with no more than Z percent errors."

Rather than try and hire a test group, or multiply a base result by a factor to indicate the scalability we decided to use scripted automated test tools. The idea with these is that you can perform the same test, under differing conditions, and receive accurate reports of the hardware and software responses.

So we tested three different products, in different market segments to see what the difference was. These were all GUI based automated testing agents.

The applications:

Paessler Web Server Stress Tool

http://www.paessler.com/order/webstress
Enterprise Edition Single User
Cost: 495.00 Euros

Pros:

• Simple GUI interface
• Flexible test suite
• Cheap

Cons:

• Harder to obtain useful report stats
• Non customizable users. The 'same' virtual user is used for all requests

---

NeoLoad

http://www.neotys.com/
Purchase:
5000 Virtual Users 26,665 Euros
unlimited Virtual Users 27,521 Euros
Rental:
5000 Virtual Users 2,152 Euros (weekly), 6,726 Euros (monthly)
Unlimited Users 2,642 Euros (weekly), 8,256 Euros (monthly)
Pros:

• Very good reporting
• Very good setup scenarios
• Flexible user setup
• Variable Data loads

Cons:

• Price
• Too customizable, can produce way too much for our needs.

---

Web Performance Suite

http://www.webperformanceinc.com/
Purchase:
5000 19995$
Unlimited 24995$
30 day leases:
5000 users, 7000$
Unlimited 8000$

Pros:

• Excellent Help system
• Detailed reporting

Cons:

• Price
• Complex interface

---

In summary:

All three products can answer the question above, each to slightly different degrees. The major difference between each product is the flexibility and scalability that they can handle.

The Paessler application provides a limited set of setup options, and as such may not produce reliable real-world results.

Both the NeoLoad and Web Performance Suite are very customizable, and produce excellent reports. Both are able to use 'real' user data to actually validate against existing records, rather than just using a single data object (like the Paessler). So would more closely emulate a real world scenario.

The biggest influencing factors here are pricing and the accuracy of results. An interesting point to mention is the option to rent some of these software solutions. As a further consideration there are open source, non GUI applications available.

They would incur much longer learning curves, and resource overheads due to someone actually having to script the test manually, but could work out a more financially viable option.