Reference User
The accuracy of load test results is important. When hardware resources are overloaded, test results can become skewed. SilkPerformer offers agent health indicators and load-test resource management out-of-the-box, but when pushing resources to the limit with GUI-based technologies (Web browser-driven, SAP or Citrix scripts), you may need to consider a more precise method.
Since hardware resources for load testing can be expensive, your goal should be to run as many virtual users as possible per machine, without overloading the agent and thereby negatively impacting test results.
Some of you may already use our CloudBurst offering and don’t need to be concerned with hardware resources. Most of you however still rely on hardware resources at your own lab.
To ensure that the results of critical tests are not falsified by overloaded load test agent machines, consider the concept of a reference user.
Concept
The reference user concept is simple. Run a single or small number of users on a dedicated machine. Usually the controller machine is sufficient for this task. If you have a spare machine, use it for this purpose. Then let your reference users run on the weakest agent machine.
Compare the response times of users on the reference agent against response times of other agents. If the response times are similar, your test was healthy and you can consider adding a few more users per load test agent machine next time.
How it’s done
Before you begin
CPU, Memory and Responsiveness are the core measures for all agent machines. Make sure that these measures remain in a healthy value range before you push agents further to their limits.
Find the peak of your agents
To quickly find out the limit of your agents, define an increasing workload that goes up slowly to the estimated total maximum of virtual users.
Chart: Testing the peak VU for 2 agent machines (lab124, lnz-spuitest) against a reference user on a weak agent machine (lab47). To make it easier to distinguish reference users from ordinary users, a copy of the usergroup was created (VUser, VReference). 2 VUsers are added to the workload every 20 seconds. The reference user starts immediately.
After the test, for each agent, drag the response times into a chart in Silk Performance Explorer. Additionally, drag the active users into the chart.
Identify the point at which the response times begin to climb for each of your agents. This is the peak that you should never exceed in future load tests.
Chart:'Trans. (busy) ok(s)' response time of agent A (green) surpasses the response time of the reference user on agent R (dark blue) at 18 active virtual users.
Chart:'Trans. (busy) ok(s)' response time of agent B (green) surpasses the response time of the reference user on agent R (dark blue) at 36 active virtual users.
Note: Make sure that you disable random thinktime in the profile settings. You want your test to be as predictable as possible.
Double check the health of your load tests
During the load test, again have your reference user(s) run on a dedicated machine. Since server load during your test may vary, the response times your users experience may also vary. Response times must however correlate with the data collected from your reference user(s).
Tips and tricks
Overruling capability settings
You can put more load on an agent than capability settings suggest by either editing the capabilities of your agent in the system settings agent pool or overruling auto-assignment by turning it off on the Workload Configuration dialog.
Make use of the begin transaction
Avoid calling resource intensive API functions in each transaction. For example, when working with Browser-Driven Load Testing (BDLT) scripts, move the BrowserStart function into the begin transaction and call only BrowserReset in the main transaction. Similar best practices exist for most technologies. Check technology-specific tutorials and documentation to learn more.