Verisium Software Testing Blog

Cloud Load Testing has recently emerged as one of the best applications of cloud computing. Cloud computing generates economies of scale and provides unprecedented access to inexpensive computing resources. While the utility of load testing is universally acknowledged, conventional load testing using in-house resources has many limitations and bottlenecks.

One of the biggest issues with load testing is that a large load test typically requires utilization of lots of hardware and this takes time to setup. Let’s consider a load test with 10,000 virtual users. If each load agent machine can handle 1,000 virtual users without performance degradation, this would mean that about 10 load agent machines are needed to run the load test. This is often very time consuming to setup. In some of the larger organizations this can take weeks. Even in smaller companies this can take a few days. Cloud load testing makes these resources available instantly. A cloud load testing tool would allow users to select the desired number of load agents and have them available within minutes.

Hardware utilization increases costs. In a typical organization the load testing may last only a few days. However, many of the load agent machines may have to be either purchased just for this reason or taken away from some other project. Purchase would clearly be expensive. Even if these machines were temporarily ‘borrowed’ from other projects, setting them up and then finally moving them back would take time. This would result in delayed projects and lower output. In cloud load testing, no such provisioning is needed. The computing resources are completely elastic and can be created and terminated on demand. If you perform cloud load testing for a few days, you only pay for those resources for that period of time.

Many websites have a wide audience that is spread geographically. In some cases this audience is global. Cloud load testing allows you to generate virtual user loads from different parts of the world. This allows more accurate simulation of the actual load. Latency for example, can increase page response times and this would only be evident in a load test where the virtual users accessing the application were geographically dispersed. If you web application is going to be accessed by users all over the world, it would be a good idea to run a load test where the virtual users are accessing the web application from all over the world. Cloud load testing tools will allow you to select users from all over the world as part of your virtual user profile.

Cloud load testing has transformed load testing and has removed many of its biggest bottlenecks and limitations. Cloud load testing provides the tools to utilize load testing to its fullest.

When we are conducting performance tests on web servers accessed by users across the globe, we cannot overlook the effects of latency on web response times. Consider a scenario where a web server is located in San Francisco and is being accessed by users in both San Francisco and Singapore. For users in San Francisco the round trip time from their machine to the web server will be negligible. However, for users in Singapore the round trip time will be significant and could introduce a delay of about 200 milliseconds (0.2 seconds).

HTTP uses TCP to transmit data to and from the web server. The TCP protocol requires a ‘handshake’ between the sender and the receiver. The TCP window size is the maximum amount of data that can be transferred without a confirmation. TCP windowing will limit throughput. Therefore, the total bandwidth of the network will not always be utilized.

Some Windows operating systems have a TCP Window Size of 64 KB. In this case a 200 millisecond latency would create a limit on the maximum effective bandwidth. This limit would roughly equal,

Maximum Effective Bandwidth = 64/0.2 = 320 KB per second.

This equals 320 KB/sec x 8 (bits per byte) to 2560 kilobits per second or 2.56 Mbps. The users in Singapore, even if they are connected to the internet at a higher bandwidth, say 10 Mbps, they will be communicating with the server in San Francisco at a significantly lower effective bandwidth. Having said all this, it is also true that many browsers use parallel TCP connections and that can overcome some of this limitation. However the parallel TCP connection technique has its limitations since it increases error rates and therefore can only enhance the effective bandwidth to a limited degree.

Why is there a latency, you might ask. The simple answer is that there is a limitation introduced by the finite speed of light. No matter how efficient anything else becomes, the speed of light will introduce a significant delay in the transmission of data across the globe. This will become more evident as we increase available internet bandwidth around the world.

As performance testers, we need to take latency into account when designing and interpreting the results of performance tests. This is particularly important for websites that have users spread around the globe.

During the performance testing of web applications, a lot of data is collected and many different metrics are calculated and displayed in real time. In this post I will try to explain exactly what these metrics signify.

Let’s assume that we have a web page that looks something like this:

newspage.html:
<html>
<body>
<A HREF=”http://news.google.com”><IMG SRC=”news-google.png” ALT=”Google News” name=”img”></A>
<A HREF=”http://news.yahoo.com”><IMG SRC=”news-yahoo.png” ALT=”Yahoo News” name=”img”></A>
<A HREF=”http://www.cnn.com”><IMG SRC=”news-cnn.png” ALT=”CNN News” name=”img”></A>
</body>
</html>

In addition let’s make the following assumptions.
• The image resource news-cnn.png is missing on the server.
• The load test is using 100 virtual users
• The load test is run for 60 seconds

If the web server is perfectly tuned and capable, each time the newspage.html is downloaded, it will trigger three additional HTTP requests attempting to download the three link images from the server. One of these is missing and therefore two of them will succeed and one will always fail.

Let’s also now look at the typical HTTP request and response headers that typically accompany such a request. The client browser will typically send something resembling the following request to the server.

Request:
GET /news/newspage.html HTTP/1.1
Host: www.mynewspages.com
Accept: *

Response:
HTTP/1.1 200 OK
Content-Type: text/html
Context-Length: 306 bytes

<html>
<body>
<A HREF=”http://news.google.com”><IMG SRC=”news-google.png” ALT=”Google News” name=”img”></A>
<A HREF=”http://news.yahoo.com”><IMG SRC=”news-yahoo.png” ALT=”Yahoo News” name=”img”></A>
<A HREF=”http://www.cnn.com”><IMG SRC=”news-cnn.png” ALT=”CNN News” name=”img”></A>
</body>
</html>

This would result in a request that would be approximately 75 bytes and a response that is 400 bytes (response headers + body).

Let’s assume that the images are 10 KB (10,000 bytes) each. Assume that each generates a response header that is 75 bytes long and a body that is 10000 bytes long resulting in a response that is 10075 bytes for two of these images and only 75 bytes for the missing image.

Thus for a typical request for the page, newspage.html, there will be 1 page downloaded and there will be 4 hits to the server (including that which obtains newspage.html itself). The 4 hits to the server will generate 4 HTTP requests. 3 of these requests will generate an HTTP 200 response which means that everything is OK. The one HTTP request to news-cnn.png will generate a HTTP 404 code. This means that although the client was able to communicate with the server the requested resource (i.e. news-cnn.png) was not found. For each

For 100 virtual users, this scenario would translate into the following web metrics (assuming a perfectly tuned web server).

Pages Downloaded per second: 100
Hits per Second: 400
HTTP Responses per Second: 400
HTTP 200: 300
HTTP 404: 100
Bytes Sent per second: 75 bytes x 4 resources x 100 virtual users = 30000 = 30 KB/sec
Bytes Received per second: (400 + 10075 + 10075 + 75) x 100 virtual users = 2062500 = 2062.5 KB/sec
Throughput = 30 + 2062.5 = 2092.5 KB/sec

The figures above assume a perfect world where everything functions perfectly. In practice the web server may not function perfectly and some requests might fail. Nevertheless, I hope that the description above gives you a basic idea of what is behind most of these numbers and how they are calculated.

How many concurrent virtual users do we need to use for load testing? This is a question that comes up very frequently. The answer is not obvious and requires a little bit of calculation and estimation. It is important to appreciate that merely averaging things would lead to inaccurate and low estimates since we will usually have time peaks that would shift the numbers upwards.

Let’s assume that we are dealing with a shopping site that has 60,000 visitors each day. It would be misleading to simply divide this by 24 and get 2,500 visitors an hour. This is because the visits would typically be concentrated around certain peak times. Assume that in the morning you have 3 times the traffic. This means that the busiest hour in the morning yields a 7500 visitor per hour traffic.

Now you need to know how the long the typical user stays on the website. If the typical user spends an average of 15 minutes (0.25 hours) on the website, we would get 60/15 = 4 visits per hour. An estimate for the number of concurrent users during load testing would be the number of users in an hour divided by the number of visits each hour. This comes out to 7500/4 = 1875. Rounding this off would yield around 2000 concurrent users. However this might represent a typical month. How about holiday season? The traffic would be higher during the holiday season. It might be twice this much. This would yield 4000 concurrent users.

Now we have an estimate. In general, it is better to estimate on the higher side as it provides some cushion. The concurrent users estimate can be made more accurate by getting better estimates for the raw numbers we have used above. Accuracy typically requires use of web server analytics tools such as Google Analytics and StatCounter. Having an accurate representation of the load your site will experience is an extremely important part of planning your load test. It permits a more accurate simulation of the typical real life scenarios.