Building quality into software is often a huge speaking point here at Northwest Cadence. We believe that testing and coding should be done simultaneously, and that everyone on the team is responsible for the quality of the software – not just the testers. One major form of testing that can go unnoticed for quite some time is how well a piece of software performs under high load. “High load” is ambiguous, but it cuts open the thought processes that make us think about what exactly that means. How many concurrent users are we expecting to have on our system? What is an acceptable performance fluctuation under high loads? When does our system start to degrade to a point of being “broken”? These questions should not be an afterthought to software. They should not be answered and tested after the functionality is in place. Questions such as these should be viewed in the same light as any other form of acceptance criteria. Let’s take a look at how we can alleviate these questions with continuous load and performance testing.
The focus of this article is to set up a test rig and to run remotely executed load tests. I will mention, and cover, many different areas that are apart of load testing, but I will keep these areas simplistic as to keep the focus on how to use Test Controllers and Test Agents for load testing.
Preparing your testing strategy
This is probably the most important step to consider when undertaking a test plan for software. Figuring out what exactly the tests should be targeting can be difficult, and the approach taken to isolate these tests can be as difficult as actually writing and performing the tests. Do we want to test the performance of different geographic locations? Do we want to figure out when the system starts to break down due to a concurrent load? Do we want to test the most active scenarios and prioritize optimizing those? The questions and different approaches can be endless. I do not want to spend a ton of time on this section, but I do want to highlight how important this part of performance and load testing is. You will not want to set up a testing rig and then be clueless as to what exactly to test. Here is an MSDN article on how to come up with a testing plan:
Preparing your machines
There are two approaches we can take when prepping machines to ready them for load testing. Essentially, we have a Test Controller that will collect and synchronize all of the performance counters collected during the test. The data can be collected either from performance monitoring counters that are out-of-box with any Windows installation, or we can install a Test Agent on the machines and use their data collectors. We will need Test Agents nonetheless to generate the load that we want to test with.
Figure 1: Visual of how the data is collected from test agents and specific servers
The advantage of having Test Agents is the intensive data collection that can occur during the tests, as well as having custom data adapters. The machines that have Test Agents installed on them can also be tagged, so the appropriate data collectors are automatically applied to the correct machines. There are also advantages to not having agents on all the machines. Depending on which data adapters are active on the Test Agents, performance can actually degrade and can totally muddle your load and performance test runs. For instance, IntelliTrace can be collected on a server which requires a lot of resources to run. These sorts of data adapters may be useful when running regression or smoke tests, but may alter your load testing results. We will go into detail on how to run the tests in both scenarios.
For my environment, I will be running Brian Keller’s Visual Studio 2012 Application Lifecycle Management Virtual Machine which is an all-in-one machine for TFS, Visual Studio, etc. This machine comes with a web application called Tailspin Toys that is already deployed to it. I am going to install and configure my Test Controller on this machine, hold my Load Test Repository, and install a Test Agent and configure it with the Test controller. For my second machine, I will use my host machine (Windows 8) and install and configure a Test Agent on it as well.
Test Controller Machine
This machine will be collecting all of the data from the Test Agents, as well as querying the servers that are specified in the load test run settings for their performance counters. We will want to install the Test Controller onto the machine and then run the Test Controller Configuration Tool which will be located in the start menu after installation. We will want to configure the Test Controller to run under a specific service account as well as be configured for Load testing.
Figure 2: Configure Test Controller to use VSALM as the database for the load test repository and set the account to a service account
In my environment, Brian is going to be the account I run my test controller under. Once everything is configured, and the account has been tested. Go ahead and hit Apply Settings, and it will apply the settings we defined here for the test controller (NOTE: You cannot have a test controller connected to a Team Project Collection as well as use it for Load Testing. This scenario is not supported). Upon applying the settings, the load test database will be created on the SQL server (if you do not have permissions to do this, there is a SQL script that can be run by a SQL Admin on the server to create the database). Article on how to do this here: http://msdn.microsoft.com/en-us/library/vstudio/ms182600.aspx). Once configured, it will succeed with warnings merely letting you know that certain accounts and permissions are set up to allow for the different scenarios. We can ignore this warning and click Close twice. When the test controller is configured, three local security groups will be created:
- TeamTestAgentService – Members of this group can connect to the Microsoft Visual Studio Test Controller service (we will want to add any accounts that the agents are running under to this group).
- TeamTestControllerAdmins – Members of this group can run tests, view/delete results, create environments, and administer test controller/agents.
- TeamTestControllerUsers – Members of this group can run tests and view results.
You will want to provision your admins and users appropriately in these groups.
Test Agent Machines
For each of your test agent machines, you will want to install the appropriately versioned Test Agent and then configure it.
Figure 3: Select the Service option and click Next
We will want to select the Service option because we are not going to be load testing coded UI tests (these, however, can be load tested as well).
Figure 4: Configure the Test Agent to use Brian as the account to run under and to register with the Test Controller we just installed.
I am going to use the same account that I used for my Test Controller purely to keep it simple in my environment. The account you use here will go into the TeamTestAgentService group on the machine that has the Test Controller installed and configured on. One more thing I would like to note is that I am not on a domain, instead I have created a shadow account for Brian on my local machine – a local account that has the same username and password as the local account on the other machine. Once finished, go ahead and click Apply Settings. This will add the correct firewall exceptions, validate settings, add the agent to the test controller, etc.
NOTE: I also add the account that the Test Controller Service is running under the Performance Monitor Users security group on each machine in my environment. This ensures that I will be able to query all of the performance counters that are available for the machines.
Once done, we can go ahead and close out of the configuration wizard.
Creating Web Performance and Load Tests
In Visual Studio, there are separate test types for Web Performance Test and for Load Test. The difference seems subtle, but they are massively different. Web Performance Tests are to create HTTP calls to replicate user behavior while using a web site. Load tests take automated tests (Web Performance Tests, Coded UI Tests, Unit Tests, etc.) and creates different scenarios to test. Because the focus of this article is to step through remotely executing load tests, I will create a simple web test and scenario to run. To start, we will need to run Visual Studio 2012 (load testing is only available in the Ultimate version) and create a new Web Performance and Load Test Project.
Figure 5: Create a new Web Performance and Load Test Project from the New Project dialog
By default, the project creates a blank WebTest1.webtest. Go ahead and open this up and click the Add Recording button on the toolbar of the web test.
Figure 6: Open the web test and click the Add Recording button
This will open up an Internet Explorer session with the Web Test Recorder toolbar on the left hand side. Go ahead and complete some user actions to be recorded and click the Stop button.
Figure 7: Record some actions and click the Stop button once finished
Once done, the web performance test will detect any parameters that you may want to bind to some sort of data source (i.e. username and password to test different user accounts).
Figure 8: The web test that was created from our previous step with the Run Tests button highlighted
Once it is done, go ahead and click the Run Test button in the toolbar to ensure that the test will succeed.
Figure 9: Web test passing after we just created it
Now we know that we have a working web test that we can start to use with a load test. We will want to create a new load test by right-clicking on the test project, selecting opening the Add menu, and then selecting the Load Test option.
Figure 10: Create a new load test for the project
This will bring up the New Load Test Wizard which will walk you through creating a scenario. Let’s walk through the wizard:
Figure 11: Click next to move past the welcome page.
We can go ahead and just click Next on the welcome page.
Figure 12: Set the name of your scenario, choose the think time profile, and choose how much think time between the test iterations
This page will allow you to choose what you want to name this scenario – I chose SimpleLoadTest for mine – as well as the scenario’s think time profile and how much time in between your test iterations. I’m going to use the recorded think times from the web test I just created as well as put a one second think time between the test iterations. You will want to do some analysis on how your services are being used, and what the normal distribution of these times would be.
Figure 13: Choose whether you want a constant load, or if you want to step load the system
I am going to choose to a constant load of 5 users for my load test. This is most likely not the case for the actual use of a website. A more real-life scenario is to have a certain amount of users for some time, and then that user concurrency loads up until peak time. You can replicate this using step loading. You can also use step loading to see when your system starts to degrade.
Figure 14: Choose your test mix model for the load test
Each of these options are how the tests are run during the load test. When selecting each one, the description in the bottom right, as well as the graphic used, will explain further what each means. I’m going to choose the Based on the total number of tests to make sure that the test distribution I use is the same at any given time during the test (this is somewhat insignificant for my scenario because I only have one web performance test).
Figure 15: Choose the test(s) that you want to add to the load test
This is where we would add the tests that we want to run during the load test. Any test that is within the project will be populated on the left. Select the test, or tests that you want to add and then click the right arrow to add it to the test. You will then be able choose the distribution of each test during the load test. Because this is the only test, keep it at 100% and click next.
I am going to skip the next two wizard screens which allows you to choose which network you want to use from the test machine as well as what sort of browser mix you want to have during the tests. Once past these, we can then then choose our counter sets. By default, the controller computer and agent computers are automatically added to the load test, even if it is a local run and not connected to test controllers. We also have the option to add a computer individually – for computers that do not have a test agent installed on them.
Figure 16: Add computers that do not have agents installed on them, or for local execution, to collect counters from.
I’m not going to add any computers here at first to see what sort of data we get back, if we only run locally with no other computers with counters being collected. The last page of the wizard is to set the run settings for the load test.
Figure 17: Set the run duration of the test as well as the sampling rate
I am going to reduce my run duration from 10 minutes down to two, so it will faster for me to run the load test. Actual load tests will want to run longer than that to get more, sustained data from the test. The sampling rate is how often the performance counters are queried by the controller for the test. Five seconds is the default, as well as the lowest this rate can go. Once done, click finish.
Figure 18: The created load test after initial wizard
This creates the load test for us. We will want to just go ahead and run it to ensure that it works, and to see what sort of counters we see. Go ahead and click the run button in the tool bar of the load test.
Figure 19: Load test results for our initial run
After running the test, we will get some data back, but not everything that we want. Note that all of counters that we see here are from my local machine because I ran a local execution of the load test. Also, we never added any other computers into the load test, so we shouldn’t expect to see anything other than the counters from our local machine. Because we may, or may not, have permissions to pull the counters from servers, we will want to set up our remote execution settings first. This way, the account the Test Controller service is running under queries will be the performance counters, and we know that account has the correct permissions to do that.
Set Up Remote Test Settings
To set up the remote test settings, right-click on the Solution Items folder and select add new item. In the Add New Item – Solution Items dialog, choose Test Settings, name it Remote.testsettings, and click the Add button. This brings up the Test Settings editor.
Figure 20: Choose Roles, select Remote execution, and type in the name of the Test Controller
We will want to select Roles from the left hand pane, then select Remote execution as our test execution method, and then type in the name of our test controller – VSALM in my environment. By default, it will choose all the agents that are connected to the controller to run the tests under. Since I originally only have my host machine as the agent, I will leave this default and click Apply and then Close.
Figure 21: Ensure that the Remote.testsettings is the active Load and Web Test Settings
After closing, we now want to ensure that our newly created test settings is currently active by right-clicking on it and selecting Active Load and Web Test Settings. Once done, we can go ahead and run the load test again, and this time the test should be occurring via the Test Agent service rather than the local Visual Studio instance.
Figure 22: Load Test results after a remote execution
There are a couple things to note now that the test was done remotely. We can see that the counters say that they are from the computer VSALM, but we know they shouldn’t be because that is the Test Controller – this is true! What this really means is that the data was collected from all of the agents by the computer VSALM. Basically, no matter how many agents are attached to the test controller, all of the agents that are running the tests – generating the load – will show as the test controller. We also want to note that the test results are no longer being saved in a local repository, they were saved, instead, to the Load Test database that was created when connecting the controller to our SQL server. Again, do not have any counters from the system under test, but we will get there shortly!
Adding System Under Test Manually
The most simplistic way to add a server under test, is to manually add it to the load test. To do so, open the load test itself, right-click on Run Settings and click Manage Counter Sets…
Figure 23: Click on Manage Counter Sets…
This then brings up the dialog to manage computers and their counter sets.
Figure 24: Add the computer, select it, and select the appropriate counter sets
I then click the Add Computer… button which gives me an empty computer node. I manually type in VSALM, check the box, and then check the appropriate sets that I would like to monitor for the system. Once again, click okay and run the Load Test.
Figure 25: Load Test results with System under Test data
Tada! We now have a load test being run by a remote test controller, the load being generated by a test agent, and data from the system under the test being collected.
Using Agents for System Under Test
Now, what if we wanted to manage what counters were collected from which servers at the controller level, rather than the load test level? We would have to install an agent onto each server and connect it up to the controller (I am going to install the agent on the same computer as the controller which I would not suggest, but I have a limited environment J). While the agent is being installed and configured, I’m going to go ahead and remove VSALM as a computer in the run settings of my load test (this is from my previous section where I manually added it to collect counter sets from). Under the Load Test menu of Visual Studio 2012, select Manage Test Controllers…
Figure 26: Select Manage Test Controllers…
By default the VSALM controller will be selected for us, and we should see the agents that are attached to the controller. From here, we want to add attributes to each machine to basically be able to segregate them out for different roles (i.e. an agent generating load and an agent that is purely used to collect SQL counter sets from). Because my environment is simple, I will label my host machine as the client machine and my web server and SQL server as the server machine.
Figure 27: Add the appropriate property to my client machine
Figure 28: Add the appropriate property to my server machine
I have now added the properties that I want to add to each agent. For more complex systems, you may want to filter down the machines more. For instance, you may want a set of properties purely for SQL servers and purely for web server counters and so on. You can get quite creative and start really isolating your load testing efforts to measure the exact scenarios that you want. After tagging our agents, we can close out of the test controller management and open up our Remote.testsettings file once again and go to the Roles section on the left hand side.
Figure 29: Add different roles and attributes for those roles
From the Roles section we can go ahead and click the Add button. This will create a role that we can name whatever we want. In the screenshot above I have created two roles – one for the client agents and one for server agents – and chose my “Client Machines” role to be the role for running the tests. Once the role is created, you can add attributes to the roles. I have added the attributes that I added to my test agents in the Test Controller Manager dialog, and then click the Preview matching test agents to see which agents have the specific attributes that are defined for the role. I made sure that the right agents were in the correct roles.
Figure 30: Choose the data and diagnostics for each role
We also have the ability to choose which data collectors are being collected for each of the roles from the Data and Diagnostics section chosen in the left hand pane. I added the ASP.NET Profiler for my server role. Once again we can click Apply, then Close, and run our load test again.
Figure 31: Load test results after using agents for every computer in the environment
We now can see that there were some threshold violations (red x’s) mainly because my computer does not have the resources to run all that stuff at once J. There are plenty of counters to pull from, and diagnosing performance issues may come from looking for the correct counters and cross-referencing them to ensure what exactly is going on with the application.
This article, again, is meant to walk through how to set up web performance and load tests and run them remotely via a Test Controller and Test Agents. Once the actual scenarios are evolved and the tests are created for them, we can see some pretty sweet results. This screenshot is the result of a testing scenario of a soon to be live application:
Figure 32: Real test results of a Load Test scenario
We can see in this test result that there are quite a bit of threshold warnings and critical violations on the System under Test. It was discovered that these are requests queuing and processing wait time counters and were due to the fact that the IIS server running the web application had a max user concurrency of 12. If small configuration changes such as these can be rooted out towards the beginning of a project, rolling out to larger audiences and ensuring that the quality of a piece of software becomes much easier.
To learn how Northwest Cadence can be a resource to you, please contact Rick Flath, Director of Business Development via telephone 425.455.1155 or email Rick.Flath@nwcadence.com.