Test Automation Tools and Rapidly Changing User Interfaces

22. July 2012 · 4 comments · Categories: Functional Testing, Scripting, Software Testing

You started using test automation tools early in the development cycle. You started to use these test automation tools to create scripts that worked well and automated a significant portion of the user interface. A few weeks later several of those test automation scripts failed. The user interface had changed and you had to spend a significant amount of time creating new test automation scripts. A few weeks later some other test automation scripts failed for the same reason. You wasted a countless number of hours working on this issue .and ending up completely frustrated by it.

The scenario described above is a well known issue faced during test automation. How does this occur? Let’s consider a web page that is described by the HTML below.

<html>
<body>
<form action=”done.aspx”>
<input id=”submit” type=”submit” value=”Submit” />
</form>
</body>
</html>

Consider a scenario where the user loads the page above and then clicks the Submit button. When you record a test automation script against the page above, many test automation tool vendors will create a script that will extract the “id” of the INPUT tag and utilize that to identify the object at runtime. This is all well and good until the “id” changes! What happens then? The script fails. When we started building a test automation tool, we had faced this problem ourselves in our past lives. As a result we designed our test automation tools to handle this issue to the extent possible. In the previous example, our test automation tool would not identify the INPUT tag above based on simply one attribute. Instead we created our own object recognition system that uses a large number of factors to identify an object. The reason is that if something simple changes e.g. one of the element’s attributes, its position in the web page etc, the test automation script will still work. What happens if the page changes to the point that it is no longer recognizable? I’ll be honest with you that in that case nobody can do much. You will have to rerecord the script. However, most user interface changes are not as drastic. Most of them are smaller changes that cause the affected page to naturally evolve over the development cycle. Many of these cases can be handled by a test automation tool that is built with this concern in mind.

There are several other ways to avoid the issues associated with rapidly changing UI. One very effective way is to keep developers involved in the test automation process and provide them with the test automation tools and your scripts so that they can themselves run the tests from time to time. This will enable them to appreciate your problems and most of them will try to work to try to avoid these problems. It will also allow the developers to discover their own defects before they reach you. Another way to avoid this issue is to spend more time in design mode as this will minimize large user interface changes during the project.

You need to use strategies to minimize having to recreate scripts. Simply using test automation tools is not enough. You need to ensure that you are using them in a manner that maximizes your productivity.

 

Share

Correlation in Load Testing

09. July 2012 · 2 comments · Categories: Load Testing, Scripting

Correlation is used to account for dynamic values in load testing. Many web applications have dynamic data that changes every time the user runs that web application. Web applications often need to track users’ interactions as they navigate through their website while preserving their state between navigations. Session ids for example are used by server engines such as ASP, ASP.NET, JSP and PHP to manage sessions. These session ids will change each time the page is loaded.

Users typically use recording mechanisms to record their scripts. In some cases, they will not be able to replay the recorded script without modifications. It doesn’t matter how good or sophisticated the load testing tool is. It is impossible to create a script recorder that can accurately record and replay every scenario. This should not be taken to mean that you should not use the recording capabilities of these load testing tools since recording scripts is a great way to create a starting script. This considerably reduces the amount of work needed to create a working script.

Here is an example to illustrate correlation and dynamic data. Consider the web page below.

<html>
<body>
<a href=”/784768″>
</body>
</html>

The link (A tag) contains a href value that is dynamic (i.e. changes every time the page is loaded). The next time the page is loaded the HTML may look something like this

<html>
<body>
<a href=”/792187″>
</body>
</html>

If you attempt to record and replay this script, unless you modify the script it will fail. The href values of the link (‘a’ tag) are dynamic and need correlation to work correctly. The only way to make this script work is to edit it and dynamically extract the href value i.e. /784768, /792187, etc. and use that value to request the subsequent page.

But hey, you might say, what about all these load testing tools that record actual browser interactions. Their users don’t need to worry about correlation and dynamically changing data since they are replaying actual user interactions (in a manner similar to functional testing tools). The problem with these kinds of load testing tools is that they will have tough time running more that around 10 or so virtual users on one machine (despite the vendor’s claim that it can run 50 or more users)! These tools will instantiate actual running instances of a web browser. This means that they also load the huge footprint of a normal web browser to accomplish a task that typically only requires a small fraction of that functionality. it is thus easy to see why once you run more than 10 or so virtual users, the machine will be completely maxed out. In this case a 1000 virtual user load will require around 100 computers, which is excessive by any standards (ordinarily, you would be able to run 1000 virtual users on single computer).

Dynamic data in load testing requires correlation to enable load testing scripts to run accurately. It is important to keep this in mind when you use recording to generate a script. If the script is unable to replay accurately, there is a good chance that this is because dynamic data is involved. The way to address this is to use correlation techniques.

Share