In today's world of e-business, your customers and business partners expect
fast performance from your Web sites. They assume that pages will download immediately
and that online transactions will be executed efficiently and accurately. Just
as in a brick-and-mortar business, where customers don't want to stand in long
lines or have their credit cards charged incorrectly, they also expect business
on the Internet to be conducted efficiently and without mistakes.
In order to compete effectively in the online consumer and B2B marketplaces,
companies should evaluate the ability of their Web-based applications to accommodate
multiple, simultaneous users viewing a Web site or engaging in online transactions.
This is typically accomplished through the purchase of an application load-testing
tool. Load-testing products simulate the experience of live users with "virtual"
users created by load-testing technology. The load-testing software can then
record the virtual users' experiences and provide this information to the client
Load-testing software is often distributive in nature. It is applied on multiple
servers running simultaneously, with each server simulating multiple virtual
users. In many cases, the testing company has developed its own proprietary
browser that can be combined with a set of instructions tailored to the testing
of each client business. The testing company maintains ongoing records of the
virtual users' experiences at the test site, including response times and errors.
Many testing companies also monitor the client Web site remotely to help diagnose
connectivity problems. The actual error messages experienced by the virtual
users may be recorded for later review. A set of logs can be created that document
each of the user experiences, and this information can later be compared with
CPU and database testing information obtained during the test by a systems administrator
or by software offered by the testing company.
One feature that load-testing companies often provide is testing the Web-based
application externally from multiple points of presence to ensure that the quality
of the provider's connectivity isn't the cause of system slowdowns. For example,
if the client network was expected to have 10 Mbps of bandwidth but consistently
experiences network slowdowns at 6 Mbps, this may signify that the network isn't
getting its expected 10 Mbps.
The next aspect of the testing is to provide performance information on the
infrastructure of the client network itself. Firewalls, routers and load balancers
may all come into play. For instance, a firewall may not have sufficient throughput
to accommodate the number of simultaneous users.
From this point forward, the testing software essentially mirrors real user
activity at the site and focuses more on the performance of the application
and the database software under stress.
In July 2002, Amplitude Research conducted an online survey to obtain the opinions
of technology-sector employees regarding their experiences with load testing
of Web-based applications. The 319 respondents represented a broad range of
technology-sector professionals, from CEOs to project managers to programmers.
Factors for Successful Load Testing
Overall, the survey results indicated generally positive evaluations of the
load-testing industry, the activities provided by vendors and specific vendors.
Respondents who were currently conducting load tests placed great importance
on the technical components of load testing while preferring an outsourced test
environment. The factors they considered most important included the following
(in decreasing order of importance):
Testing at different speeds. The reason for testing at different
speeds is to determine if slower connections use more resources. This may
reduce the number of virtual users who may simultaneously visit a Web site.
Testing on different browsers. This is not really a load-testing
issue, but it can offer important insight into the error-free performance
of Web-based applications on different browsers.
The ability to support complicated scripts to simulate user experiences.
To simulate a real user experience, the load-testing company needs to script
the test and provide instructions to the testing browsers. This experience
needs to closely resemble the transactions performed by real users at the
Additionally, our survey results indicated that people wanted easier-to-understand
reports. Customers gave the lowest comparative rating on the survey to the user-friendliness
of the reports offered by current testing tools. Given the importance of this
activity, vendors should invest considerable time to develop concise and easily
understandable reporting methods of the testing data.
Customers who reported the highest levels of overall satisfaction with their
load-testing applications were those who believed that their products did a
good job of predicting real-world performance and, secondarily, provided easy-to-understand
reports. Price was not a statistically significant predictor of satisfaction
with a vendor, nor was the total IT budget of the respondent's company. Overall,
individuals currently using load-testing tools placed greater importance on
the various activities provided by vendors than did individuals who were not
presently conducting load tests.
In evaluating a load-testing product, it is important to closely examine the
load-testing software's ability to simulate the real user experience at your
Web site. This requires evaluating your individual needs and verifying that
the testing company can adequately meet your needs through scripting, monitoring
and other relevant activities. The more information that the testing software
can accumulate about your Web site and the user experiences, the more likely
it is that any problem areas can be identified.
It's also important to evaluate what reporting information and advice the testing
company provides after the test is completed. You should ask to see a copy of
a sample report and examine what information is actually provided. Additionally,
it's important to evaluate the experience of the load-testing company and whether
the type of projects it has completed are similar to your requirements. The
recommendations it provides after the test is completed will often be the key
"takeaway" from the load test.