Share →
Buffer

Building quality into software is often a huge speaking point here at Northwest Cadence. We believe that testing and coding should be done simultaneously, and that everyone on the team is responsible for the quality of the software – not just the testers. One major form of testing that can go unnoticed for quite some time is how well a piece of software performs under high load. “High load” is ambiguous, but it cuts open the thought processes that make us think about what exactly that means. How many concurrent users are we expecting to have on our system? What is an acceptable performance fluctuation under high loads? When does our system start to degrade to a point of being “broken”? These questions should not be an afterthought to software. They should not be answered and tested after the functionality is in place. Questions such as these should be viewed in the same light as any other form of acceptance criteria. Let’s take a look at how we can alleviate these questions with continuous load and performance testing.

The focus of this article is to set up a test rig and to run remotely executed load tests. I will mention, and cover, many different areas that are apart of load testing, but I will keep these areas simplistic as to keep the focus on how to use Test Controllers and Test Agents for load testing.

Preparing your testing strategy

This is probably the most important step to consider when undertaking a test plan for software. Figuring out what exactly the tests should be targeting can be difficult, and the approach taken to isolate these tests can be as difficult as actually writing and performing the tests. Do we want to test the performance of different geographic locations? Do we want to figure out when the system starts to break down due to a concurrent load? Do we want to test the most active scenarios and prioritize optimizing those? The questions and different approaches can be endless. I do not want to spend a ton of time on this section, but I do want to highlight how important this part of performance and load testing is. You will not want to set up a testing rig and then be clueless as to what exactly to test. Here is an MSDN article on how to come up with a testing plan:

Preparing your machines

There are two approaches we can take when prepping machines to ready them for load testing. Essentially, we have a Test Controller that will collect and synchronize all of the performance counters collected during the test. The data can be collected either from performance monitoring counters that are out-of-box with any Windows installation, or we can install a Test Agent on the machines and use their data collectors. We will need Test Agents nonetheless to generate the load that we want to test with.

Figure 1: Visual of how the data is collected from test agents and specific servers

The advantage of having Test Agents is the intensive data collection that can occur during the tests, as well as having custom data adapters. The machines that have Test Agents installed on them can also be tagged, so the appropriate data collectors are automatically applied to the correct machines. There are also advantages to not having agents on all the machines. Depending on which data adapters are active on the Test Agents, performance can actually degrade and can totally muddle your load and performance test runs. For instance, IntelliTrace can be collected on a server which requires a lot of resources to run. These sorts of data adapters may be useful when running regression or smoke tests, but may alter your load testing results. We will go into detail on how to run the tests in both scenarios.

My Configuration

For my environment, I will be running Brian Keller’s Visual Studio 2012 Application Lifecycle Management Virtual Machine which is an all-in-one machine for TFS, Visual Studio, etc. This machine comes with a web application called Tailspin Toys that is already deployed to it. I am going to install and configure my Test Controller on this machine, hold my Load Test Repository, and install a Test Agent and configure it with the Test controller. For my second machine, I will use my host machine (Windows 8) and install and configure a Test Agent on it as well.

Test Controller Machine

This machine will be collecting all of the data from the Test Agents, as well as querying the servers that are specified in the load test run settings for their performance counters. We will want to install the Test Controller onto the machine and then run the Test Controller Configuration Tool which will be located in the start menu after installation. We will want to configure the Test Controller to run under a specific service account as well as be configured for Load testing.

Figure 2: Configure Test Controller to use VSALM as the database for the load test repository and set the account to a service account

In my environment, Brian is going to be the account I run my test controller under. Once everything is configured, and the account has been tested. Go ahead and hit Apply Settings, and it will apply the settings we defined here for the test controller (NOTE: You cannot have a test controller connected to a Team Project Collection as well as use it for Load Testing. This scenario is not supported). Upon applying the settings, the load test database will be created on the SQL server (if you do not have permissions to do this, there is a SQL script that can be run by a SQL Admin on the server to create the database). Article on how to do this here: http://msdn.microsoft.com/en-us/library/vstudio/ms182600.aspx). Once configured, it will succeed with warnings merely letting you know that certain accounts and permissions are set up to allow for the different scenarios. We can ignore this warning and click Close twice. When the test controller is configured, three local security groups will be created:

  • TeamTestAgentService – Members of this group can connect to the Microsoft Visual Studio Test Controller service (we will want to add any accounts that the agents are running under to this group).
  • TeamTestControllerAdmins – Members of this group can run tests, view/delete results, create environments, and administer test controller/agents.
  • TeamTestControllerUsers – Members of this group can run tests and view results.

You will want to provision your admins and users appropriately in these groups.

Test Agent Machines

For each of your test agent machines, you will want to install the appropriately versioned Test Agent and then configure it.

Figure 3: Select the Service option and click Next

We will want to select the Service option because we are not going to be load testing coded UI tests (these, however, can be load tested as well).

Figure 4: Configure the Test Agent to use Brian as the account to run under and to register with the Test Controller we just installed.

I am going to use the same account that I used for my Test Controller purely to keep it simple in my environment. The account you use here will go into the TeamTestAgentService group on the machine that has the Test Controller installed and configured on. One more thing I would like to note is that I am not on a domain, instead I have created a shadow account for Brian on my local machine – a local account that has the same username and password as the local account on the other machine. Once finished, go ahead and click Apply Settings. This will add the correct firewall exceptions, validate settings, add the agent to the test controller, etc.

NOTE: I also add the account that the Test Controller Service is running under the Performance Monitor Users security group on each machine in my environment. This ensures that I will be able to query all of the performance counters that are available for the machines.

Once done, we can go ahead and close out of the configuration wizard.

Creating Web Performance and Load Tests

In Visual Studio, there are separate test types for Web Performance Test and for Load Test. The difference seems subtle, but they are massively different. Web Performance Tests are to create HTTP calls to replicate user behavior while using a web site. Load tests take automated tests (Web Performance Tests, Coded UI Tests, Unit Tests, etc.) and creates different scenarios to test. Because the focus of this article is to step through remotely executing load tests, I will create a simple web test and scenario to run. To start, we will need to run Visual Studio 2012 (load testing is only available in the Ultimate version) and create a new Web Performance and Load Test Project.

Figure 5: Create a new Web Performance and Load Test Project from the New Project dialog

By default, the project creates a blank WebTest1.webtest. Go ahead and open this up and click the Add Recording button on the toolbar of the web test.

Figure 6: Open the web test and click the Add Recording button

This will open up an Internet Explorer session with the Web Test Recorder toolbar on the left hand side. Go ahead and complete some user actions to be recorded and click the Stop button.

Figure 7

Figure 7: Record some actions and click the Stop button once finished

Once done, the web performance test will detect any parameters that you may want to bind to some sort of data source (i.e. username and password to test different user accounts).

Figure 8: The web test that was created from our previous step with the Run Tests button highlighted

Once it is done, go ahead and click the Run Test button in the toolbar to ensure that the test will succeed.

Figure 9: Web test passing after we just created it

Now we know that we have a working web test that we can start to use with a load test. We will want to create a new load test by right-clicking on the test project, selecting opening the Add menu, and then selecting the Load Test option.

Figure 10: Create a new load test for the project

This will bring up the New Load Test Wizard which will walk you through creating a scenario. Let’s walk through the wizard:

Figure 11: Click next to move past the welcome page.

We can go ahead and just click Next on the welcome page.

Figure 12: Set the name of your scenario, choose the think time profile, and choose how much think time between the test iterations

This page will allow you to choose what you want to name this scenario – I chose SimpleLoadTest for mine – as well as the scenario’s think time profile and how much time in between your test iterations. I’m going to use the recorded think times from the web test I just created as well as put a one second think time between the test iterations. You will want to do some analysis on how your services are being used, and what the normal distribution of these times would be.

Figure 13: Choose whether you want a constant load, or if you want to step load the system

I am going to choose to a constant load of 5 users for my load test. This is most likely not the case for the actual use of a website. A more real-life scenario is to have a certain amount of users for some time, and then that user concurrency loads up until peak time. You can replicate this using step loading. You can also use step loading to see when your system starts to degrade.

Figure 14: Choose your test mix model for the load test

Each of these options are how the tests are run during the load test. When selecting each one, the description in the bottom right, as well as the graphic used, will explain further what each means. I’m going to choose the Based on the total number of tests to make sure that the test distribution I use is the same at any given time during the test (this is somewhat insignificant for my scenario because I only have one web performance test).

Figure 15: Choose the test(s) that you want to add to the load test

This is where we would add the tests that we want to run during the load test. Any test that is within the project will be populated on the left. Select the test, or tests that you want to add and then click the right arrow to add it to the test. You will then be able choose the distribution of each test during the load test. Because this is the only test, keep it at 100% and click next.

I am going to skip the next two wizard screens which allows you to choose which network you want to use from the test machine as well as what sort of browser mix you want to have during the tests. Once past these, we can then then choose our counter sets. By default, the controller computer and agent computers are automatically added to the load test, even if it is a local run and not connected to test controllers. We also have the option to add a computer individually – for computers that do not have a test agent installed on them.

Figure 16: Add computers that do not have agents installed on them, or for local execution, to collect counters from.

I’m not going to add any computers here at first to see what sort of data we get back, if we only run locally with no other computers with counters being collected. The last page of the wizard is to set the run settings for the load test.

Figure 17: Set the run duration of the test as well as the sampling rate

I am going to reduce my run duration from 10 minutes down to two, so it will faster for me to run the load test. Actual load tests will want to run longer than that to get more, sustained data from the test. The sampling rate is how often the performance counters are queried by the controller for the test. Five seconds is the default, as well as the lowest this rate can go. Once done, click finish.

Figure 18: The created load test after initial wizard

This creates the load test for us. We will want to just go ahead and run it to ensure that it works, and to see what sort of counters we see. Go ahead and click the run button in the tool bar of the load test.

Figure 19: Load test results for our initial run

After running the test, we will get some data back, but not everything that we want. Note that all of counters that we see here are from my local machine because I ran a local execution of the load test. Also, we never added any other computers into the load test, so we shouldn’t expect to see anything other than the counters from our local machine. Because we may, or may not, have permissions to pull the counters from servers, we will want to set up our remote execution settings first. This way, the account the Test Controller service is running under queries will be the performance counters, and we know that account has the correct permissions to do that.

Set Up Remote Test Settings

To set up the remote test settings, right-click on the Solution Items folder and select add new item. In the Add New Item – Solution Items dialog, choose Test Settings, name it Remote.testsettings, and click the Add button. This brings up the Test Settings editor.

Figure 20: Choose Roles, select Remote execution, and type in the name of the Test Controller

We will want to select Roles from the left hand pane, then select Remote execution as our test execution method, and then type in the name of our test controller – VSALM in my environment. By default, it will choose all the agents that are connected to the controller to run the tests under. Since I originally only have my host machine as the agent, I will leave this default and click Apply and then Close.

Figure 21: Ensure that the Remote.testsettings is the active Load and Web Test Settings

After closing, we now want to ensure that our newly created test settings is currently active by right-clicking on it and selecting Active Load and Web Test Settings. Once done, we can go ahead and run the load test again, and this time the test should be occurring via the Test Agent service rather than the local Visual Studio instance.

Figure 22: Load Test results after a remote execution

There are a couple things to note now that the test was done remotely. We can see that the counters say that they are from the computer VSALM, but we know they shouldn’t be because that is the Test Controller – this is true! What this really means is that the data was collected from all of the agents by the computer VSALM. Basically, no matter how many agents are attached to the test controller, all of the agents that are running the tests – generating the load – will show as the test controller. We also want to note that the test results are no longer being saved in a local repository, they were saved, instead, to the Load Test database that was created when connecting the controller to our SQL server. Again, do not have any counters from the system under test, but we will get there shortly!

Adding System Under Test Manually

The most simplistic way to add a server under test, is to manually add it to the load test. To do so, open the load test itself, right-click on Run Settings and click Manage Counter Sets…

Figure 23: Click on Manage Counter Sets…

This then brings up the dialog to manage computers and their counter sets.

Figure 24: Add the computer, select it, and select the appropriate counter sets

I then click the Add Computer… button which gives me an empty computer node. I manually type in VSALM, check the box, and then check the appropriate sets that I would like to monitor for the system. Once again, click okay and run the Load Test.

Figure 25: Load Test results with System under Test data

Tada! We now have a load test being run by a remote test controller, the load being generated by a test agent, and data from the system under the test being collected.

Using Agents for System Under Test

Now, what if we wanted to manage what counters were collected from which servers at the controller level, rather than the load test level? We would have to install an agent onto each server and connect it up to the controller (I am going to install the agent on the same computer as the controller which I would not suggest, but I have a limited environment J). While the agent is being installed and configured, I’m going to go ahead and remove VSALM as a computer in the run settings of my load test (this is from my previous section where I manually added it to collect counter sets from). Under the Load Test menu of Visual Studio 2012, select Manage Test Controllers…

Figure 26: Select Manage Test Controllers…

By default the VSALM controller will be selected for us, and we should see the agents that are attached to the controller. From here, we want to add attributes to each machine to basically be able to segregate them out for different roles (i.e. an agent generating load and an agent that is purely used to collect SQL counter sets from). Because my environment is simple, I will label my host machine as the client machine and my web server and SQL server as the server machine.

Figure 27: Add the appropriate property to my client machine

Figure 28: Add the appropriate property to my server machine

I have now added the properties that I want to add to each agent. For more complex systems, you may want to filter down the machines more. For instance, you may want a set of properties purely for SQL servers and purely for web server counters and so on. You can get quite creative and start really isolating your load testing efforts to measure the exact scenarios that you want. After tagging our agents, we can close out of the test controller management and open up our Remote.testsettings file once again and go to the Roles section on the left hand side.

Figure 29: Add different roles and attributes for those roles

From the Roles section we can go ahead and click the Add button. This will create a role that we can name whatever we want. In the screenshot above I have created two roles – one for the client agents and one for server agents – and chose my “Client Machines” role to be the role for running the tests. Once the role is created, you can add attributes to the roles. I have added the attributes that I added to my test agents in the Test Controller Manager dialog, and then click the Preview matching test agents to see which agents have the specific attributes that are defined for the role. I made sure that the right agents were in the correct roles.

Figure 30: Choose the data and diagnostics for each role

We also have the ability to choose which data collectors are being collected for each of the roles from the Data and Diagnostics section chosen in the left hand pane. I added the ASP.NET Profiler for my server role. Once again we can click Apply, then Close, and run our load test again.

Figure 31: Load test results after using agents for every computer in the environment

We now can see that there were some threshold violations (red x’s) mainly because my computer does not have the resources to run all that stuff at once J. There are plenty of counters to pull from, and diagnosing performance issues may come from looking for the correct counters and cross-referencing them to ensure what exactly is going on with the application.

Conclusion

This article, again, is meant to walk through how to set up web performance and load tests and run them remotely via a Test Controller and Test Agents. Once the actual scenarios are evolved and the tests are created for them, we can see some pretty sweet results. This screenshot is the result of a testing scenario of a soon to be live application:

Figure 32: Real test results of a Load Test scenario

We can see in this test result that there are quite a bit of threshold warnings and critical violations on the System under Test. It was discovered that these are requests queuing and processing wait time counters and were due to the fact that the IIS server running the web application had a max user concurrency of 12. If small configuration changes such as these can be rooted out towards the beginning of a project, rolling out to larger audiences and ensuring that the quality of a piece of software becomes much easier.

  

To learn how Northwest Cadence can be a resource to you, please contact Rick Flath, Director of Business Development via telephone 425.455.1155 or email Rick.Flath@nwcadence.com.

Print Friendly
  • ijaz saeed

    great work !!!

  • Anuj Sharma

    Excellent Article and very helpfull. Can you tell me how to add more than one load agent to the test?
    I want to make rig with 3 agents and one controller.
    -Anuj

  • James Tupper

    Follow the steps outlined in the section titled “Test Agents” for each machine. After the agents are added, ensure they have the same tag you used for the machines that will be used to run the tests from in the remote settings (I used “clientMachine=true” in my example above).

  • Anuj Sharma

    Thanks for your reply. However i have done it by installing test agent setup on 2-3 diffrent machines.
    Now i am facing challange in collecting data while VU are ‘cool down stage’. Currently what is happening my all the users are not gradually exiting from test. My test duration is 1 Hr 15 Minutes and cool down period is 20 minutes(250 VUs). But after test duration my test is not recording the transactions during the cool down period.
    If during the test i have 1500 successfull Login transactions and i didn’t got any failures, then at the end Logout transactions should be 1500. But this is not happening, I am getting transactions summary as :
    Login 1500, Home Page 1480, Dashboard 1450……logout 890 transactions.
    Do you have any clue how to over come this hurdel.
    setting is set to
    Anuj

  • James Tupper

    Both the warmup and cooldown portions are not meant to be used for gathering data. Since you want to see how your system under test performs under certain circumstances, these periods would throw your metrics off considerably. If you are however trying to run load tests based on the number of iterations, this blog can help you achieve that: http://blogs.msdn.com/b/billbar/archive/2006/02/16/533216.aspx.

  • Danny

    Hello,

    Hopefully you can help. When assigning an agent to a remote server (not an agent and controller as per your example) the agent shows correctly. When running a load test it shows as running however I cannot see any data from it.

    It does not appear in the list of computers and there are no errors related to it. If I change its role to execute tests then it is fine but I don’t want the server to execute its tests.

    Any ideas?

  • James Tupper

    Are you generating the load locally, then? The controller collects the data from the agents during test runs. So, without a controller you will need to manually add in the remote machine in to gather the counter sets during a test run.

  • Danny

    Hello,

    I am not running locally.

    I have 7 Agents, 5 Workers,2 on two Web Server (Load Balanced) and a controller all on separate machines.

    The only way I have been able to make sure the 2 agents on the web servers do not execute the test is to go to the scenario and set the Agents to run the tests and also set the 2 web server agents weight to 0.

    The roles does not seem to work for monitoring the web servers.

  • James Tupper

    Did you tag each agent with appropriate attributes? You can use the attributes to denote which computers should be generating the load, rather than setting the agents weight to 0. The section above titled ‘Using Agents for System Under Test’ highlights how to do this.

  • asaf

    Hi,

    I have VS2012 Ultimate Upgrade 3.

    I recorded WebTest and generated a code (using VS2012 tool).

    I want to “Expose” the TestMethod that was created to the MS Test Runner so I will be able to run it (and later otehr scenario’s) automatically.

    My code looks like :

    [TestClass]

    public class WebClientLoginLogoutCoded : WebTest {

    public WebClientLoginLogoutCoded() {

    this.PreAuthenticate = true;

    }

    [TestMethod]

    public override IEnumerator GetRequestEnumerator() {…}

    and When I try to complie it, I get the error :

    UTA007: Method GetRequestEnumerator defined in class ReadyMind_Load.WebClientLoginLogoutCoded does not have correct signature. Test method marked with the [TestMethod] attribute must be non-static, public, does not return a value and should not take any parameter. for example: public void Test.Class1.Test(). Additionally, return-type must be Task if you are running async unit tests. Example: public async Task Test.Class1.Test2().

    how can I solve my problem (I have few directions) :

    1. should I change the GetRequestEnumerator() signature, if yes to what?

    2. should I change one (or both) the attribute, if yes to what?

    3. is there another way that I can “Expose”/”Declare” the test method to ms test runner?

    Thanks

  • Pankaj

    Hey guys, I followed this article (pretty great), but running into No such host is known issue all the time.

    I’m trying to run the tests that I created in VS 2012 Ultimate and have TestController and TestAgent on the same machine. Any info would help.

  • Josh

    You may need to add your test user account to the Performance Log and Monitor Users groups in the machines you wanna collect information from.

  • Josh

    Hi, I have a question when you run a load test/webtest locally or remotely from an agent, the page response times that you collect are the actual page response times or do they include the communication time between the controller and agent machine.

    Basically when you submit a request
    VS starts the test and sends the request to the controller (T1) + Controller sends the request to the agent (T2) + Agent receives it and forwards to the server (T3) + Server sends the response back to the agent (T4) + Agent sends the information to the controller (T5) + Controller sends the statistics to the VS (T6)

    The response time value we see in our webtset or load test results include which of them?
    Ideally we would like to see T3 as the request time and T4 as the response time or T3+T4 as the turn around time. But I am suspicious o it is not clear that those statistics may also include forwarding times between VS, controller and agents. Do you have an idea about this issue?

  • James Tupper

    The request/response times would be T3/T4, respectively. The Test Controller handles the orchestration of the tests. The tests are sent to the agents, and the agents run the tests and record the data. Every 5 seconds, by default (this can be changed), the controller will ping all the agents and servers under test that you have to retrieve the necessary data.

    Does this answer your question?

  • James Tupper

    Where are you seeing the no such host is known issue? Is it when you’re running the tests, or when you’re trying to set up a test rig in order to run the performance tests?

  • Dan

    For “System Under Test” metrics, since most people will have the Test Controller and Test Agent separate from systems being tested (Web Server and SQL DB) what is required to pull in this information during load testing?
    Do I simply add my test controller account to the Performance Monitor User group on these servers? Do I need to install test agent on these servers and configure it for performance counter collection?
    Any information on this would be greatly appreciated.
    Thank you.

  • James Tupper

    The “System Under Test” metrics are just an out of the box report that pools the data together for any machine that was added to the test directly. The test controller account being in the Performance Monitor User group on those servers is the only permission requirement needed in order to gather these metrics because the counters already exist on any Windows OS. Merely adding the machines to your test will have the controller poll the counters from those machines.
    Alternatively, you can install test agents on those machines. This requires a different approach as you now have to tag the servers and clients appropriately in order to differentiate which machines will be generating the load, and which will be the systems under test. It’s truly a matter of personal preference as to whether, or not, you want to install test agents on the systems that are being tested.

  • Ljubica

    Absolutely!!!
    Can you please tell me how can I remove the limited 250 concurrent user limit for Load Testing, in Visual Studio Ultimate 2013

  • James Tupper

    Thanks, Ljubica!

    The 250 concurrent user limit for Load Testing was removed for Visual Studio 2012. There no longer is a limit for number of virtual users, and a single machine/agent can run up to 2000 virtual users.

    Here’s a blog post noting the removal of the 250 concurrent user limit: http://blogs.msdn.com/b/howie_hillikers_blog/archive/2013/06/26/visual-studio-2012-removal-of-the-250-virtual-user-limit-for-load-testing.aspx

  • Ljubica

    Thanks James,

    The next issue, Configuration of the Test Controller failed. Failed to restart service.
    I’m not able to restart the Test Controller service, also I’m not able to restart it manually from the Services-section.
    Thanks.

  • James Tupper

    I’ve sent you an email pertaining to the screenshots you sent me!

  • Kunal Raje

    Hi James,

    Can you please guide me for
    installation and configuring Controllers and Agents for Web Performance and
    Load test using VSTS 2013 Ultimate? Also, Kindly let me know Is there a
    need for separate machines for Controllers and Agents OR one machine will do the
    same test? I need to perform these tests for our internal Share Point
    application.

    Thanks and Regards,

    Kunal Raje

  • James Tupper

    This particular article walks through how to install and configure a controller and agents to work with web performance and load testing. Here is the official MSDN article on using test controllers and agents for load testing: http://msdn.microsoft.com/en-us/library/ms243155.aspx

    You can put one controller and one agent on a single machine; however, there is a limitation of one test agent per machine. Each test agent can generate 2000 virtual users.

  • saurabhshri27

    What is the relation between constant user count and run duration of the test. If i increases the duration, page count will also increases. Does VS restarts virtual users after their simulation has been completed. It simulates each user in a loop until the specified time period elapses. Please advice. Do we have any link (msdn or other) on the same.

  • James Tupper

    It really depends on the test mix model you use. If you use the model that will run the tests based off of a percentage of virtual users, then those tests will be run over and over again until the duration of the load test has completed. However, if you choose a test mix model based on how many times that test is run in a given time interval, then you would see a lower number of tests being run on those virtual users.
    Here’s an MSDN link that dives a bit more into detail about test mix models: http://msdn.microsoft.com/en-us/library/bb514186.aspx

  • saurabhshri27

    Thanks James. It helped me a lot.:-)

  • Guest

    Can we run the load test irrespective of time duration, for example, if i am running tests for 25 users then test will automatically stops, once all the users finished their scripts. please help on this ?

  • saurabhshri27

    Can we run the load test irrespective of time duration, for example, if i am running tests for 25 users then test will automatically stops, once all the users finished their scripts. please help on this ?
    I ran with test iteration 25 and constant user 25. Is this correct approach ? It ran successfully but key indicator graph is showing constant user 25 till last second. I think users should decreases in the graph as tests gets completed. Please advice.

  • James Tupper

    The specific scenario you have laid out isn’t quite one that is easy to do with the tools as they are. What is the reasoning behind why you want the users to stop as their tests become complete? Would it not be easier to merely run web performance tests individually via a script if you are only trying to run the test once?

  • saurabhshri27

    I also want 25 users parallel, how it would be possible with web performance test. What about test iteration 25 with constant users 25 ?

  • saurabhshri27

    Can we insert checkpoints in the scripts. For example In load tests. we get results for each URL. If site is using same URL for many different pages, that pages lumped together into single URL in the result. In QALoad, it was possible to create “checkpoints” in the script, and that is how we were able to see the performance for each page, regardless of URL. Can we achieve this by VSTS ? Please help.

  • saurabhshri27

    Can we change the order of Page Results in Load Test. We get results according to “Avg. Page Time (sec)” in descending order in “Page Results”. Can we change this order, i mean according to sequence they occurred in web test. Please advice.

  • James Tupper

    You are not able to change the default summary page for load tests themselves. Here is a link that goes over the load test results overview page: http://msdn.microsoft.com/en-us/library/bb385841.aspx

    However, all of the data is available for you to report off of – if you’re doing remote execution, the results should be stored in the load test database.

  • James Tupper

    As far as I know this is not possible. Load tests are, simply put, ways to run other types of tests in large quantities. Web performance tests are the type I use in this blog, and these tests are based off of URL calls. If your application uses the same URLs and a web performance test is not picking up those URLs, you may have to create unit tests that will let you control the granularity of your HTTP calls.

  • saurabhshri27

    Thanks for your comments, I am using “Reporting Name” property of web test URL. In this way, we can get the results irrespective of URL.

  • Biswajeet Tripathy

    Hi,
    I am getting the below error message after completing the load test using VSTS 2012 ultimate and not able to view load test summary and counters data.
    “Could not read result repository. The Connection String property has not been initialized.Will display the run time result instead of repository result.”

    But already the load test results store has been setup in manage test controller window.Please help me on this.

  • Thank you for sharing such useful information. Your article
    helped me to understand career prospects in software testing industry. Testing
    is the part of the software development. It assists the programmers and developers
    to create stunning application catering the requirement of individuals. Software
    testing training institutes in Chennai

  • vignesh

    Excellent blogs!!!You have sharing to lot of information…hadoop training chennai

  • I got the basics of mobile
    testing. I want to know the approaches used in automate mobile testing http://www.fita.in/mobile-testing-training-in-chennai/

  • Martin Hatch

    Great article, but got a question. When I try to execute any test using “Remote Execution” I get “No such host found”. The Controller connects to VS fine (and lists all of the Agents, all of which say “Ready”). Setup has 3 boxes (1 Controller with Agent, 1 Agent, 1 with Visual Studio)

  • tin acosta

    Question, What is the impact of using a test rig in load testing? what is the main purpose of using test rig compared to the local run settings only?

  • James Tupper

    Thanks, Martin! Is it failing prior to the load test starting, or does it start failing after the initialization? This may be due to the agents not being able to find the target website versus your computer not being able to connect to the test controller.

  • James Tupper

    I’m not 100% sure what you mean by the “impact of using a test rig in load testing.” Are you talking about performance hits on the agents themselves, or about the target servers? Being able to use a test rig allows you to scale out the number of virtual users (as you would be limited by the physical properties of your local machine). You can also test from various geographical locations if you so wish. The list goes on, but the main point is that it allows you to diversify and scale out your load test as needed.

  • tin acosta

    Thank you for your reply! 🙂 what i meant about the impact of using a test rig is that what does it does to the load test that a local run settings cannot offer?

  • tin acosta

    What is also the effect of using test rig in the system under test computers?

  • Inna Radchenko

    Hi! 🙂
    Thank you so much for this great article.
    I’m really new at this. Also getting a similar error “failed to queue test run, no such host is known” and it happens one second after running the load test. In “Manage test controllers” the agents are ready and it seems like everything is connected correctly. What to do? Please help!

  • James Tupper

    Are you able to reach the target website if you were to log into one of the agent machines?

  • Chandan Gupta

    Hi,We have an option to generate code for Performance test created. Do we have the same for corresponding Load test wizard added to it? I want to have the code version so as to parameterise configurable items with different values.

  • Inna Radchenko

    Yes…

  • James Tupper

    Is the test controller able to reach the SQL database that is holding the load test database?

  • lucy

    Hi,

    Thanks for your valuable post…and what difference between white box testing and black box testing on manual and ETL..http://www.thinkittraining.in/manual-and-qtp-qc

  • Vikas

    I am using VS2010 ultimate to test the performance of the application
    a) Is there a limit on number of scenarios to be executed.
    b)Is this possible to configure just one machine as controller as well as Agent.
    c)Is this possible to increase/decrease the number of users dynamically while scripts are running.

  • Ayyappa sabbisetty

    I have a web test with custom web test plug in which I am
    able to execute successfully with local execution. When I execute same test with remote execution
    (Controller and agent set up), my test is getting failed with below error. Can someone
    help me to fix this issue?

    Request failed: Exception in
    PreWebTest event: Could not create instance of rule class ‘CheckDataJMS’
    :Exception has been thrown by the target of an invocation.

    at Microsoft.VisualStudio.TestTools.WebTesting.WebTestFrameworkHelperMethods.CreateInstance(Type
    type, PropertyCollection properties)

    at
    Microsoft.VisualStudio.TestTools.WebTesting.WebTestPluginReference.CreateInstance()

    at
    Microsoft.VisualStudio.TestTools.WebTesting.DeclarativeWebTest.OnPreWebTest(PreWebTestEventArgs
    e)

    at Microsoft.VisualStudio.TestTools.WebTesting.WebTest.InvokePreWebTest()

    at Microsoft.VisualStudio.TestTools.WebStress.WebTestCaseVariation.Start()

  • Bharath

    Hi All,

    I am suppose to perform load and performance testing on an web application which is initiated from a server. There will be 100’s of users connecting to it.

    I don’t want to install any framework or tools which will do load testing on the client or the server side… It should be installed and monitored ONLY by a single system (which is with me). Is there any tool or framework that can do this ?

    If there is any manual script that I need to do, please let me know in Python because I am familiar with it.

  • I am not sure that I understand @disqus_TSZeapP5ih:disqus . The above process does not require the install of any tools or frameworks onto the systems under test. They are external to the system under test. You have Agents that execute the load tests against the system under test and they are installed separately from the target system.
    If you want to have a limited, small, number of simulated users you can easily run the load tests from Visual Studio on your local system. You will be limited by bandwidth and CPU capabilities of that single system.
    Agents, especially using the Visual Studio Online Load Tests scaling solution can support more than a million simultaneous users.

  • Bharath

    Thanks for the reply @MrHinsh:disqus, Is it necessary for me to install these “Agents” on all of the client systems ?
    My requirement is that these; 100 clients will be accessing a server and running a web app. I want to monitor the performance of these client-side browser activities which is the performance tests and on the server-side I need to check the load as well as the performance. This monitoring of load and performance tests needs to be done on a single machine which is with me. Ca this be done without installing any agents on server-side or client-side ?

    Also I have written certain test cases in python w/ selenium. Can you suggest me tools or frameworks which can run with python ? Thanks

  • As long as your account (the one running the tests) has administrator rights to the target machines then you can read the performance counters remotely and don’t need an agent installed.

    There is built in integration to run Selenium tests from MTM and have them pushed to environments and run there. That is a more complicated setup than just load testing from Visual Studio, but it works well.

  • Jignesh Patel

    I ran a load test using visual studio online account. In one test of only few clicks on website it has measured only user loads and % processor time and nothing else. Its not showing any other graph and no errors.

    In other test of many pages, it has done same and measured only user load and %processor time, only difference in this test is % Processor time has crossed critical treshold value.

    I am not able to find any document on why its not measuring other values and also what is meaning of having more than 90% processor time. As on my Azure Server (where i have website hosted and i am measuring performance off) it says processor never crossed above 10%.

    Can any one help please?

  • Kiran Chavada

    visual studio load test in which PC processor time calculate server or my local pc?

  • Daniel Nash

    Hi, I’m having the same error, “failed to queue test run, no such host is known”. I can access the controller server, and when I manage controllers, I am able to connect to the database (Test connection succeeded). It seems to be happening almost immediately, with “Test run could not be executed” showing in the error message. I’ve been stuck on this for days…

  • Sachi Williamson

    Hi, Daniel! Would you be interested in hopping on a Skype meeting for 30 minutes with the team here at Northwest Cadence to see if we can help with this? Let us know what time works best for you at clientservices@nwcadence.com.

  • Daniel Nash

    Thanks heaps for the offer, but managed to solve it yesterday. Was a poblem with my hosts file

  • Shyamsundar S

    Hi,
    I have generated WebTest to Code .cs file and I did some modification in the generated code. Now I need to convert it back to webTest file. How to generate a webtest file from .cs file?
    I need to upload the webtest file to Azure for Availability Test.

  • MarloNielson3

    Useful blog post , BTW , if others is requiring to merge PDF or PNG files , my colleague encountered a tool here http://goo.gl/HPbDYM

  • Ranjan m

    i configured test controller, test agent but i am not getting the controller in Remote execution of VS2012, how to display controller? please help me…

  • rajesh Chaganti

    HI I have a peformance lab setup wherein I have a framework which has the unit test. Now, I have included the unit test into load test but I see that one agent is not taking more than 100 user load. Can anyone please suggest like any settings specially are designed to handle this. I’m using VSTS 2013 Ultimate.

  • Don

    I am Testing a simple application using above VS 2015.
    1. Launch the Browser and navigate to App
    2. Login to the app.
    3. Create a reservation (this is our business)
    4. Logout of App

    Recorded the script with user User01 (with its password) and able to play back without any issue and able to find the reservation in our Database (that shows there is new reservation created by the user User01)

    Used the same script and added a data source that is .csv file with User02 & User03. Did (in Local.test settings)setting (one run per data source row) and executed Web test and also Load test. ( Tried both Web Test and Load test)
    Execution is fine and I see User02 & User03 been used in Run1 and Run2 in VS 2015. But when I check our reservation Database it shows there are two new reservations created by the user User01 (the one I recorded script with) and this is an issue.
    It should actually have created two new reservations with two users namely User02 & User03 (that are in my .csv file).
    Since 5 days I have been looking into Microsoft Msdn documentation and pluralsight videos and I am not missing anything on my end.
    Could you let me know, what I am missing or what I should be adding.

  • Thanks for sharing this!

  • excellent post

  • WeiseGuy

    I know this was 2 years ago…but can you help me? I have the exact same issue you mentioned.

    Client is my dev box.
    Test Controller is on a remote box
    Test Agent is on a third remote box.

    All three can ping each other, all the setups worked perfectly fine. I even added the host entries to each box just in case, even though they were all resolving fine with nslookup/ping.

    I’m using VS2017 on my client, and ensured I used the applicable controller and agent installers when I set them up manually. My SQL DB is on the controller box and testing connection from the client works fine. I can use SQL Server Manager from the client, from the controller, and from the agent and connect to that DB.

    I’m at a loss, no idea what to do now.

  • WeiseGuy

    Did you ever get this fixed? I have the exact same setup with the exact same issue.

  • WeiseGuy

    If you’re still around and checking this thread, any way you can help me out with my setup?