Performance Testing, LoadRunner Tips&Tricks

General: Planning for Load Testing

Monitors: Memory Glossary

2007-08-01T13:33:00.000+08:00

This is a compilation of memory-related counters. Do feel free to revert or comment should you have additional information on the counter.

Available Bytes

Process Working Set growth becomes constrained when Available Bytes <>

Pool Nonpaged Bytes

The system’s non-pageable (fixed) memory.

Pool Paged Resident Bytes

The OS‘s pageable memory that is currently resident in RAM.

System Code Resident Bytes

[Coming soon]

System Driver Resident Bytes

Total System Resident Bytes = Pool Nonpaged Bytes + Pool Paged Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes + System Cache Resident Bytes.

System Cache Resident Bytes

The current amount of RAM used for the file cache.

Page Faults/sec

Can be a grossly misleading number. Page Faults/sec = “soft” Transition Faults + application file Cache Faults + demand zero faults + hard page faults.

Page Reads/sec

This counter measures the number of requests to the I/O manager to retrieve pages of memory from the disk. Despite the name of this counter, it measures requests, not pages; a request can be for more than a single page.
Equivalent to hard page fault rate.

Page Writes/sec

This counter measures the number of requests to the I/O manager to write pages of memory to the disk. Again, each request can be for more than a single page.
Updated “dirty” pages must be flushed to disk before they can be reused by a different application.

Pages Input/sec

This counter measures the number of pages read from the disk per second. Combing this counter with the Page Reads/sec counter can tell you how many pages are retrieved per request.
Calculate the bulk paging rate: Pages Input/sec / Pages Read/sec

Pages Output/sec

This counter measures the number of pages written to the disk per second. Combining this counter with the Page Writes/sec counter can tell you how many pages are written per request.
Try to limit Pages Input/sec + Pages Output/sec to 10 – 20% of total disk bandwidth, if possible. Disk bandwidth absorbed for paging operations is unavailable for application processes.

Pages/sec

This counter measures the total number of pages read and written to the disk. This counter represents the sum of Pages Input/sec and Pages Output/sec. Using this counter along with the Disk Bytes/sec counter from the Physical Disk Object, you can determine what portion of the data transferred to the disk is due to memory access, and what portion is due to file system access.

Committed Bytes

Represents virtual memory pages backed in either RAM or secondary storage (paging files). Calculate a virtual: real memory contention index = Committed Bytes / Total RAM. Consider adding RAM when this ration starts to approach 2:1.

Commit Limit

Maximum number of virtual memory pages that can be allocated without extending the paging file(s).

% Committed Bytes in Used

Committed Bytes / Commit Limit. Consider adding RAM when consistently > 70% on a server.

Cache Bytes

Actually, the system address space working set, but includes the file cache. The sum of Pool Paged Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes + System Cache Resident Bytes.

Transition Faults/sec

“Soft” page faults resolved without having to access the disk.

Cache Faults/sec

Normal application file I/O operations are diverted to use the paging subsystem. Each file cache fault leads to a physical disk read I/O operation.

Demand Zero Faults/sec

The rate at which applications require brand new pages.

Write Copies/sec

Private Copy on Write pages from shared DLLs.

Pool Paged Bytes

Calculate a virtual: real memory contention index = Pool Paged Resident Bytes / Pool Paged Bytes. Compared to Page Reads/sec to anticipate real memory bottlenecks.

General: Planning for Load Testing - Monitor Setup

2007-07-30T22:25:00.000+08:00

The setting up of monitors poses some challenge and I advised you to cater one or two days to implement the monitors. LoadRunner features agentless monitoring (which other products do too). However, in order to achieve such feature it needs the SUT (System Under Test) to be configured properly.

This “configure properly” becomes sort of “out-sourced” to the system admin team and if problem arises, the blame of product issue will be reduced. Depending on the type of monitors you are implementing, you will require time and effort to clear ports and justify your case to the security or the network team before the actual implementation is possible.

This is where the initial gathering of information is important in Soliciting Requirements. With the architecture diagram, you can relate to the security or network team on the team of installation of agents or opening of ports. But of course this will come under scrutiny if you do not provide sufficient convincing information to them.

This is also the part of load test where you start to work with even more people from other teams, such as, system, network, database, security or whatever titles they are given.

Also, take note that for different applications or servers, configuration has to be done before the tool can collect monitoring information from them. Examples like perfmon for Windows, rstatd for UNIX, JMX or JMS for WebLogic.

Point to take note at this stage, every organization has different requirements in their infrastructure and for this, you need to respect that and be sensitive to it.

Related Topics

General: Planning for Load Testing

General: Planning for Load Testing - Protocols

2007-07-25T10:04:00.000+08:00

Load testing is about capturing the traffic sent by the application to the server. And using the captured traffic, the tool scripts out APIs to generate the traffic, which is been emulated during replay and sent back to the server.

Through understanding the Application Design, you should be able to determine the type of protocol that is been communicated. As I pointed in previous articles , LoadRunner focuses a lot on protocol and therefore the need to know this information is critical.

To be able to read the logs and find meaningful information out of it, I would recommend you to acquire as much knowledge as possible. For example, HTTP, the replay log merely displays the transmission of the HTTP requests in text. From there, you will know information such as authentication headers or the body that was sent.

I cannot emphasize again that do not associate functional testing as load testing. Therefore, if you are from the functional testing background, please be sure to acquire more knowledge on the protocol that you are working on before proceeding.

Related Topics

General - Protocols

Understanding Memory: Virtual Addressing (Illustration)

2007-07-23T09:26:00.001+08:00

The following illustration explains the process of virtual addressing when a program is loaded into memory. For detailed explanation, refer to "Understanding Memory: Virtual Addressing".

Executable program image file loads into memory.
Logical memory address range of the application is divided into fixed-size chunks caled pages.
Page mapping in Virtual Memory is dynamic.
Frequently referenced logical addresses tend to reside in physical memory.
Infrequently referenced pages are relegated to paging files on secondary disk storage.

General: Planning for Load Testing - Application Design

2007-07-22T17:37:00.000+08:00

This is the point that the load tester must understand how the application works. When we refer to how it works, it’s not about just clicks on links and buttons, or the calendar will pop up. It goes deeper such as the following but not limited to the list:

What is the type of communication between the client and server?
Is the application client-side?
Is there client-side activities involved in the entire application?
What is the authentication mechanism?
Does the application allow multiple login of a single user?
How does the application maintain a session?
How does the load balancer distribution the load?
What are the parameters been sent to the server?

Basically, this is actually gathering the technical details of the application unlike the initial Soliciting Requirements which comprises of the “big picture” information.

Without knowing how it works and you start clicking the links and buttons, you are bound for a failing load test at the script level. Also if you were intending to seek assistance from the paid support, chances is that they will ask you if you know the application well enough such as protocol used, client-side activities, and so on so forth.

This information should be consulted with the application developer as he/she is the one that developed it and will be the most knowledgeable amount the performance test team.

However, take note that you may bump into an inexperienced developer of the application, which might end up providing wrong information and wasting everyone’s time and effort. My advice is to be sure of the technology you are working with before approaching them.

Another thing to note, from a consultant or vendor company perspective, I do recognize that the load tester may have trouble conveying such messages or questions to your client’s development team. The time for them to answer your questions might not be as fruitful as writing another two modules to them. But bare minimum is that you must be able to translate and convince them the importance for the need to know the application design.

Related Topics

General: Planning for Load Testing
General - Content Page

General: Planning for Load Testing - Soliciting Requirements

2007-07-21T00:53:00.000+08:00

Prior to this stage, I always give a “sales talk” of what we do for a living and also to get them excited (if they are not). Gathering and soliciting accurate requirements will save you lots of unnecessary work.

Be sure to get hold of the requirements to monitor such as acceptable Transaction Response Time or server utilization. Clients need to be clear of what they want before you can proceed to achieve (unless is you who proposes the benchmark). For example, if CIO wants the response time for a search to be fewer 10 seconds will be a good metric to achieve. This is also the exit criteria as per se for the load test project.

They may also like to know the utilization of the servers under load and not really concerned with the response time. Usually, these are applications that are already problematic and it’s what they are hiring you to dig the cause of the bottleneck.

Always get an overview of the architecture which includes all components that inhabitant it. You can leave out the number of switches or routers if they do not provide much usage except routing and separating networks. It is important to get information like number of servers and what is housed in it, load balancers, firewall and databases. This will aid you in discussion and knowing what machines to be monitored.

Assist the clients in understanding which business process may raise performance issues or they see it’s criticality in the business competitive edge. With this, it can translate into the scripts that will be used in the load test.

Related Topics

General: Planning for Load Testing
General - Content Page

General: Planning for Load Testing

2007-07-19T11:01:00.000+08:00

After writing a couple of posts for the past few months, I would like to touch on the basic understanding of using LoadRunner in Performance Testing. This will provide an overview how you should facilitate the entire load testing process and be aware of the requirements that surfaces. If course, this is not limited to other load testing tools because the concepts are the same.

There are few areas to be aware or trained in order to have the load test proceed with minimal difficulties. I’ve broken them down into the following: namely, soliciting requirements, application design, protocols, monitor setup, monitoring, analyzing and recommendations.

Soliciting Requirements

Related Topics

Understanding Memory: Virtual Addressing

2007-07-15T11:55:00.000+08:00

Virtual memory is a feature supported by most advanced processors. Hardware support for virtual memory includes hardware mechanism to map from logical (virtual) memory address that application programs reference to physical (real) memory hardware addresses.

When an executable program’s image file is first loaded into memory, the logical memory address range of the application is divided into fixed-size chunks called pages. These logical pages are them mapped to similar-sized physical pages that are resident in real memory.

This mapping is dynamic in that frequently referenced logical addresses tend to reside in physical memory (also know s RAM, real memory, or main memory), while infrequently referenced pages are relegated to paging files on secondary disk storage. The active subset of virtual memory pages associated with a single process’s address space currently resident in RAM is known as the process’s working set, because they are the active pages referenced by the programs as it executes.

Virtual addressing is designed to be transparent to application programs, allowing them to be written without regard to specific real memory limitations of this or that computer. Virtual addressing even makes it possible for an executing program to reference an address space that is larger than the amount of physical memory installed on a particular computer. Virtual addressing allows a programmer to exploit what looks like a virtually infinite-sized computer memory where each individual process can address up to 4 GB of virtual addresses. As of the above, performance issues still exists when the OS attempts to reference more memory locations than can actually fit inside real memory.

Virtual memory system works well because executing programs seldom require all the associated code and data areas they allocate to be resident in physical memory concurrently in order to run. With virtual memory, only the active pages associated with a program’s current working set remain resident in real memory. On the other hand, virtual memory systems can run very poorly when the working sets of active processes greatly exceed the amount of physical RAM that the computer contains. It is important to understand the logical and physical memory usage to diagnose performance problems arising from real memory being over-committed.

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

Content Page - General
Page Fault Resolution
Page Fault Resolution (Illustration)
Performance Concerns
Virtual Memory Shortage Alerts
Available Bytes
LRU
System Working Set
Detecting Memory Leaks
Measuring Memory Utilization

General: Understanding Memory - Page Fault Resolution (Illustration)

2007-07-15T11:15:00.001+08:00

Illustration of Page Fault Resolution. To better understand how Page Fault occurs with respect to the illustration, refer to article, "General: Understanding Memory - Page Fault Resolution".

Fig 1: Page Fault Resolution

The sequence of Page Fault resolution as follows.

Thread attempts to reference page in memory.
Page not resident in real memory thus not found.
Hardware interrupt occur to resolve page fault.
ISR gains control to validate the referenced address.
Locate page on secondary storage.
Copy page into available free page in real memory.
Resume thread execution cycle.

The time taken for Page Fault Resolution thus equivalent from Step [3] to [6]. Having mentioned that, if memory is the performance bottleneck with the number of Page Faults increases over time, the application/system may experience delay due to the time needed to perform Page Fault Resolution pointed in Step [3] to [6].

Related Topics

Page Fault Resolution
Performance Concerns
Virtual Memory Shortage Alerts
Available Bytes
LRU
System Working Set
Detecting Memory Leaks
Measuring Memory Utilization

Basics: Network Bottlenecks

2007-07-14T23:18:00.000+08:00

At the network level, many things can affect performance. The bandwidth (the amount of data that can be carried by the network) tends to be the first culprit checked. Assuming you have determined that bad performance is attributable to the network component of an application, there is more likely cause of bad network performance than network bandwidth. The most likely cause of bad network performance is the application itself and how it is handling distributed data and functionality.

The overall speed of a particular network connection is limited by the slowest link in the connection chain and the length of the chain. Identifying the slowest link is difficult and may not even be consistent: it can vary at different times of the day or for different communication paths. A network communication path lead from an application through a TCP/IP stack (which adds various layers of headers, possibly encrypting and compressing data as well), then through the hardware interface, through a modem, over a phone line, through another modem, over to a service provider’s router, through many heavily congested data lines of various carrying capacities and multiple routers with different maximum throughputs and configurations, to a machine at the other end with its own hard interface, TCP/IP stack and application. A typical web download route is just like this. In addition, there are dropped packets, acknowledgements, retries, bus contention, and so on.

Because so many possibilities causes of bad network performance are external to an application, one option you can consider including in an application is a network speed testing facility that reports to the user. This should test the speed of data transfer from the machine to various destinations: to itself, to another machine on the local network, to the Internet Service Provider, to the target server across the network, and to any other destinations appropriate. This type of diagnostics report can tell users that they are obtaining bad performance from something other than your application. If you feel that the performance of your application is limited by the actual network communication speed, and not by other (application) factors, this facility will report the maximum possible speeds to your user.

Latency

Latency is different from the load-carrying capacity (bandwidth) of a network. Bandwidth refers to how much data can be sent down the communication channel for a given period of time and is limited by the link in the communication chain that has the lowest bandwidth. The latency is the amount of time a particular data packet takes to get from one end of the communication channel to the other. Bandwidth tells you the limits within which your application can operate before the performance become affected by the volume of data being transmitted. Latency often affects the user’s view of the performance even when bandwidth isn’t a problem.

In most cases, especially Internet traffic, latency is an important concern. You can determine the basic round-trip time for a data packets from any two machines using the ping utility. This utility provides a measure of the time it takes a packet of data to reach another machine and be returned. However, the time measure is for a basic underlying protocol (ICMP packet) to travel between the machines. If the communication channel is congested and the overlying protocol requires re-transmissions (often the case for Internet traffic), one transmission at the application level can actually be equivalent to many round trips.

It is important to be aware of these limitations. It is often possible to tune the application to minimize the number of transfers by packing data together, caching and redesigning the distributed application protocol to aim for a less conversational mode of operation. At the network level, you need to monitor the transmission statistics (using the ping and netstat utilities and packet sniffers) and consider tuning any network parameters that you have access to in order to reduce re-transmissions.

TCP/IP Stacks

The TCP/IP stack is the section of code that is responsible for translating each application-level network request (send, receive, connect, etc.) through the transport layers down to the wire and back up to the application at the other end of the connection. Because the stacks are usually delivered with the operation system and performance-tested before delivery (since a slow network connection on an otherwise fast machine and fast network is pretty obvious), it is unlikely that the TCP/IP stack itself is a performance problem.

In addition to the stack itself, stacks include several tunable parameters. Most of these parameters deal with transmission details beyond the scope of the book. One parameter worth mentioning is the maximum packet size. When your application sends data, the underlying protocol breaks the data into packets that are transmitted. There is an optimal size for packets transmitted over a particular communication channel, and the packet size actually used by the stack is compromise. Smaller packets are less likely to be dropped, but they introduced more overhead, as data probably has to be broken up into more packets with more header overhead.

If your communication takes place over a particular set of endpoints, you may want to alter the packet sizes. For a LAN segment with no router involved, the packets can be big (e.g. 8KB). For a LAN with routers, you probably want to set the maximum packet size to the size the routers allow to pass unbroken. (Routers can break up the packets into smaller ones; 1500 bytes is the typical maximum packet size and the standard for the Ethernet. The maximum packet size is configurable by the router’s network administrator.) If your application is likely to be sending data over the Internet and you cannot guarantee the route and quality of routers it will pass through, 500 bytes per packet is likely to be optimal.

Network Bottlenecks

Other causes of slow network I/O can be attributed directly to the load or configuration of the network. For example, a LAN may become congested when many machines are simultaneously trying to communicate over the network. The potential throughput of the network could handle the load, but the algorithms to provide communication channels slow the network, resulting in a lower maximum throughput. A congested Ethernet network has an average throughput approximately one third the potential maximum throughputs. Congested networks have other problems, such as dropped network packets. If you are using TCP, the communication rate on a congested network is much slower as the protocol automatically resends the dropped packets. If you are using UDP, your application must resend multiple copies for each transfer. Dropping packets in this way is common for the Internet. For LANs, you need to coordinate closely with network administrators to alert them to the problem. For single machines connected by a service provider, suggesting improvements. The phone line to the service provider may be noisier than expected: if so, you also need to speak to the phone line provider. It is also worth checking with the service provider, who should have optimal configurations they can demonstrate.

Dropped packets and re-transmissions are a good indication of network congestion problems, and you should be on constant lookup for them. Dropped packets often occur when routers are overloaded and find it necessary to drop some of the packets being transmitted as the router’s buffer overflow. This means that the overlying protocol will request the packets to be resent. The netstat utility lists re-transmission and other statistics that can identify these sorts of problems. Re-transmissions may indicate that the maximum packet size is too large.

DNS Lookup

Looking up network address is an often-overlooked cause of bad network performance. When your application tries to connect to a network address such as foo.bar.somthing.org (e.g. downloading a webpage from http://foo.bar.something.org), your application first translates foo.bar.somthing.org into a four-byte network IP address such as 10.33.6.45. This is the actual address that the network understands and uses for routing network packets. The is this translation works is that your system is configured with some seldom-used files that can specify this translation, and a more frequently used Domain Name System (DNS) server that can dynamically provide you with the address from the given string. DBS translation works as follows:

The machine running the application sends the text string of the hostname (e.g. foo.bar.something.org) to the DNS server.
The DNS server checks its cache to find an IP address corresponding to that hostname. If the server does not find an entry in the cache, it asks its own DNS server (usually further up the Internet domain-name hierarchy) until ultimately the name is resolved. (This may be by components of the name being resolved, e.g. first .org, then something.org, etc, each time asking another machine as the search request is successively resolved.) This resolved IP address is added to the DBS server’s cache.
The IP address s returned to the original machine running the application.
The application uses the IP address to connect to the desired destination.

The address lookup does not need to be repeated once a connection is established, but any other connections (within the same session of the application or in other session s at the same time and later) need to repeat the lookup procedure to start another connection.

You can improve this situation by running a DNS server locally on the machine, or on a local server if the application uses a LAN. A DNS server can be run as a “caching-only” server that resets its cache each time the machine is rebooted. There would be little point in doing this if the machine used only one or two connections per hostname between successive reboots. For more frequent connections, a local DNS server can provide a noticeable speedup to connections. Nslookup is useful for investigating how a particular system does translations.

Related Topic

Tools: Determine the Windows patches installed using Windows 2003 Resource Kit

Tools: Monitoring Unix Resource Script ver 1.0

2007-07-11T09:23:00.001+08:00

Due to the past experiences working in my organization, having the problem of not able to monitor the Unix system resources due to whatever reasons (rstatd configuration, dynamic ports, security reasons or lazy administrators) had always been bothering us.

Our team came up with the simple initiative to monitor using the same concept as perfmon. We requested (outsourced) one of our interims to search for unix monitoring scripts and modify it to collect data periodically. And here we are, special thanks to the interim, we finally got the first-cut of the scripts.

Copy and paste the below script starting from here ==================================================================

#!/bin/bash
# (C) 2006 Mark Boddington, http://www.badpenguin.co.uk/

# (C) 2007 Modified by Hwee Seong, http://loadrunnertnt.blogspot.com
# Licensed under the GNU GPL Version 2.
# ***** Configuration *****
# set LOG to the directory you want to write the performance data to.
# set HDD to the number of had disks in your machine.
# NOTE : Do not change the variable LIMIT
LOG=/home/Admin/report
HDD=2
LIMIT=0
genStat()
{
test=$1
SLEEP=$2
filename=$(date +%Y%m%d%H%M%S)
for((num=0;num> ${LOG}/vm.${filename}.log &
sleep $SLEEP
done
if [[ LIMIT -eq test ]]
then
#VM csv
echo Date,Time,Procs in Run Queue,Procs Blocked,Virtual Memory Used,Idle Memory,Buff Memory Used,Cache Memory Used,Swap In,Swap Out,Blocks Received,Blocks Sent,System Interrupt,Context Switches/sec,Cpu User Time,Cpu System Time,Cpu Idle Time,Cpu Waiting Time> ${LOG}/vm.${filename}.csv
cat ${LOG}/vm.${filename}.log awk '{for(l=1;l<17;l++)>> ${LOG}/vm.${filename}.csv
fi
}
case $1 in
run)
genStat $2 $3
;;
*)
echo -e ":::: Usage ::::"
echo -e "$0 run 5 10 : Collect stats for 5 times with 10 seconds interval"
echo -e "Note : Change the first parameter for the number of stats"
echo -e " Change the second parameter for the number of seconds between each stats"
;;
esac

==================================================================

The original script was developed by Mark Boddington and could be obtained from http://www.badpenguin.co.uk

The modified script allows the interval between each collection and the number of times the collection is made. However, this script lacks the ability to set the output path and it is created based on vmstat. Also, you will have to estimate the completion time for the scenario to complete.

To use the script, copy the quoted content (above) and name it MonitorPerformance.sh. Then in the directory that the script exists, type the following:

\MonitorPerformance.sh 5 10

Where 5 is the number of times the collection is made and 10 is the interval between each collection in seconds.

Once the collection is complete, import the file in analysis. Our settings in Analysis are as follows.

This is a minimal start for us and we intend to further enhance the script to allow the following (provided we can squeeze some time out):

1. Changing of the output path

2. Include iostat and netstat.

Please feel free to use the script and we welcome suggestions, modifications and enhancement to the scripts to make it a better tool.

(simple tools making lifes easier)

Scripts: Starting a new transaction during iterations

2007-07-09T13:45:00.000+08:00

Recently one of my load testing projects required us to monitor the time taken to upload a file into the application. This sounds simple and straight forward with the implementation of the lr_start_transaction and the lr_end_transaction. However, the catch is that the action block be iterated 10 times and within the 10 iterations, one of the iteration is to upload a different file of different size (thus causing different response time), which they want to measure.

What we did was implement a if-then-else to check the file and perform a lr_start_transaction when the script is uploading the different file and end the transaction with the lr_end_transaction when the submission completes. The snippet of the code is as followed:

if (!strcmp(lr_eval_string("{file}"), lr_eval_string("{set_file}")))

{

lr_start_transaction("tx_check_upload_different_file");

}

else

{

lr_start_trasaction("tx_tx_check_same_upload_file");

}

...

\\other codes

...

...
if (!strcmp(lr_eval_string("{file}"),lr_eval_string("{set_file}")))

{

lr_end_transaction("tx_check_upload_different_file",LR_AUTO);

}

else

{

lr_end_trasaction("tx_check_same_upload_file",LR_AUTO);

}

The code detects the if the uploaded file is different and will start tracking the timing for the upload transaction. Hope it's useful to you.

Related Topics

Scripts: Duplicating Files on the Fly

Scripts: Step Download Timeout

Scripts: VIEWSTATE

Scripts: Auto or Manual Correlation?

Scripts: Remove Think Time

Scripts: Set Debug Mode in Script

Scripts: Replay Failure – Use Full Extended Log!

Basics: Memory Bottlenecks

2007-07-08T16:02:00.000+08:00

Maintaining watch directly on the system memory (RAM) is not usually that helpful in identifying performance problems. A better indication that memory might be affecting performance can be gained by watching for paging of data from memory to the swap files. Most current OS have a virtual memory that is made up of the actual (real) system memory using RAM chips, and one or more swap files on the system disks. Processes that are currently running are operating in real memory. The OS can take pages from any of the processes currently in real memory and swap them out to disk. This is known as paging. Paging leaves free space in real memory to allocate to other processes that need to bring in a page from disk.

Obviously, if all the processes currently running can fit into real memory, there is no need for the system to swap out any pages. However, if there are too many processes to fit into real memory, paging allows the system to free up system memory to run more processes. Paging affects system performance in many ways. One obvious way is that if a process has had some pages moved to disk and the process becomes run-able, the OS has to pull back the pages from dusk before that process can be run. This leads to delays in performance. In addition, both CPU and the disk I/O spend time doing the paging, reducing available processing power and increasing the load on the disks. This cascading effect involving both the CPU and I/O can degrade the performance of the whole system in such a way that it maybe difficult to even recognize that paging is the problem. The extreme version of too much paging is thrashing, in which the system is spending so much time moving pages around that it fails to perform any other significant work. (The next step is likely to be a system crash).

As with run-able queues (see CPU section), a little paging of the system does not affect the performance enough to cause concern. In fact, some paging can be considered good. It indicated that the system’s memory resources are fully utilized. But at the point where paging becomes a significant overhead, the system is overloaded.

Monitoring paging is relatively easy. On UNIX, the utilities vmstat and iostat provide details as to the level of paging, disk activity and memory levels. On Windows, the performance monitor has categories to show these details, as well as being able to monitor the system swap files.

If there is more paging than is optimal, the system’s RAM is insufficient or processes are too big. To improve this situation, you need to reduce the memory being used by reducing the number of processes or the memory utilization of some processes. Alternatively, you can add RAM. Assuming that it is your application that is causing the paging (Otherwise, either the system needs an upgrade, or someone else’s processes may also have to be tuned), you need to reduce the memory resources you are using.

When the problem is caused by a combination of your application and others, you can partially address the situation by using process priorities (see the CPU “section”). The equivalent to priority levels for memory usage is an all-or-nothing option, where you can lock process in memory. This option is not available on all systems and is more often applied to shared memory than to processes, but nevertheless, it is useful to know. If this option is applied, the process is locked into real memory and is not paged out at all. You need to be aware that using this option reduces the amount of RAM available to all other processes, which can make overall system performance worse. Any deterioration in system performance is likely occurring at heavy system load, so make sure you extrapolate the effect of reducing the system memory in this way.

Related Topic

Products: SilkPerformer (a comparison with LoadRunner)

2007-07-06T16:21:00.000+08:00

SilkPerformer is a load testing tool from Borland, originally Segue. SilkPerformer have almost all features that LoadRunner have. Therefore, you may like to read about the product LoadRunner here before proceeding further on this article. The SilkPerformer suite consists of the Workbench, Silk TrueLog and Silk Performance Explorer. The entire load test implementation is the same where the load generators have to be installed at the machines. For the sales talk by the vendor, click here.

What SilkPerformer offer is it handles load test in project approach. Using this approach, each load test is stored as a project with various settings of profiles and workloads. This is advantageous when managing the load test in a customer or project perspective where you can track the projects accordingly. Furthermore, within a project, for every load test run, a result directory for each run is created. This is useful in tracking the number of runs that were conducted. In this way, there is a structural approach in managing the tests that were conducted. Both features mentioned above are not available in LoadRunner.

SilkPerformer have another cool feature, which is resource management of the load generator. By defining a pool of load generators, the load tester is able to have an overview of the resources that is participating in the load test. This is almost similar to the Resource Pool in Performance Center. This is not available in LoadRunner.

During the execution, SilkPerformer allows monitoring of the resources in terms of CPU usage in the Load Generator. This is beneficial when differing problem in resources on the Load Generator or the SUT itself. Furthermore, feature allows capacity planning for load generators. SilkPerformer also allows monitoring on the progress of the vusers and load test in terms of percentage that displays the progress in percentage of the vusers. This is beneficial in estimating the duration of the load test and forecast the end time of the load test.

While performing the evaluation, I found that the parameterization feature offered in SilkPerformer is not as user-friendly as compared to LoadRunner. The manipulation of the parameterization is not as straightforward such as defining the parameter and reusing the parameter unlike LoadRunner where a replace function can easily achieve the parameterization.
For iteration, SilkPerformer have to define the iteration number in the script. For pacing a LoadRunner feature, is not offered in the SilkPerformer.

SilkPerformer provides Rendezvous feature in terms of serialisation and synchronisation. This is useful if a certain load model needs to be emulated during the test (better than LoadRunner). However, the downside is it requires inclusion of the API which is not as convenient as LoadRunner in implementing a Rendezvous point.

For replay, the Silk TrueLog is a powerful tool in analysing the replay as compared to LoadRunner Runtime Viewer. If offers replay of the screens and providing views of the data in and out from the client and server.

SilkPerformer allows monitoring similar to LoadRunner. However, LoadRunner provides more features in the monitoring perspective. SilkPerformer requires additional logging for non-default monitors unlike LoadRunner, which automatically logs the data when the counter is been added into the monitor list. Also, SilkPerformer is restricted to a defined set of custom monitors namely JMX, SNMP, Perfmon and Rexec unlike LoadRunner, which provides more monitors.

SilkPerformer utilises Silk Performance Explorer for analysing. However, it’s capability to merge, manipulate, handle external data is not as flexible and robust as LoadRunner Analysis. The analysing feature is an important component used by the Performance Team to determine problem causes.

SilkPerformer have the similar license mechanism as LoadRunner, which is bounded by a single host. However it features an additional mechanism that allows the option to "check-out" license to the installed application. This is useful in an organisation context in managing our performance test in different environments.

Related Topics

Basics: Disks Bottlenecks

2007-07-04T20:46:00.000+08:00

In most cases, applications can be tuned so that disk I/O does not cause any serous performance problems. But if, application tuning, you find that disk I/O s still causing a performance problem; your best bet may be to upgrade the system disks. Identifying whether the system has a problem with disk utilization is the first step. Each system provides its own tools to identify disk usage (Windows has a performance monitor, and UNIX has the sar, vmstat, iostat utilities.) At minimum, you need to identify whether the paging is an issue (look at disk-scan rates) and assess the overall utilization of your disks (e.g. performance monitor on Windows, output from iostat –D on UNIX). It may be that the system has a problem independent of your application (e.g. unbalanced disks), and correcting this problem may resolve the problem issue.

If the disk analysis does not identify an obvious system problem that is causing the I/O overhead, you could try making a disk upgrade or a reconfiguration. This type of tuning can consist of any of the following:

Upgrading to faster disks
Adding more swap space to handle larger buffers
Changing the disk to be striped (where files are striped across several disks, thus providing parallel I/O. e.g. with a RAID system)
Running the data on raw partitions when this is shown to be faster.
Distributing simultaneously accessed files across multiple disks to gain parallel I/O
Using memory-mapped disks or files

If you have applications that run on many systems and you do not know the specification of the target system, bear in mind that you can never be sure that ant particular disk is local to the user. There is a significant possibility that the disk being used by the application is a network-mounted disk. This doubles the variability in response times and throughput. The weakest link will probably not even be constant. A network disk is a shared resource, as is the network itself, so performance is hugely and unpredictably affected by other users and network load.

Disk I/O

Do not underestimate the impact of disk writes on the system as a whole. For example, all database vendors strongly recommend that the system swap files be placed on a separate disk from their databases. The impact of if not doing so can decrease database throughput (and system activity) but an order of magnitude. This performance decreases come from not splitting I/O of two disk-intensive applications (in this case, OS paging and database I/O).

Identifying that there is an I/O problem is usually fairly easy. The most basic symptom is that things take longer than expected, while at the same time the CPU is not at all heavily worked. The disk-monitoring utilities will also tell you that there is a lot of work being done to the disks. At the system level, you should determine the average peak requirements on the disks. Your disks will have some statistics that are supplied by the vendor, including:

The average and peak transfer rates, normally in megabytes (MB) per seconds, e.g. 5MB/sec. Form this, you can calculate how long an 8K page takes to be transferred from disk, and for example, 5MB/sec is about 5K/ms, so an 8K page takes just under 2ms to transfer.

Average seek time, normally in milliseconds (ms). This is the time required for the disk head to move radially to the correct location on the disk.

Rotational speed, normally in revolutions per minutes (rpm), e.g. 7200rpm. From this, you can calculate the average rotational delay in moving the disk under the disk-head reader, i.e., the time taken for half a revolution. For example, for 7200rpm, one revolution takes 60,000ms (60 seconds) divided by 7200rpm, which is about 8.3 ms. So half a revolution takes just over 4ms, which is consequently the average rotational delay.

This list allows you to calculate the actual time it takes to load a random 8K page from the disk, this being seek time + rotational delay + transfer time. Using the examples given in the list, you have 10 + 4 + 2 = 16 ms to load a random 8K page (almost an order of magnitude slower than the raw disk throughput). This calculation gives you a worst –case scenario for the disk-transfer rates for your application, allowing you to determine if the system is up to the required performance. Note that if you are reading data stored sequentially in disk (as when reading a large file), the seek time and rotational delay are incurred less than once per 8K page loaded. Basically, these two times are incurred only at the beginning of opening the file and whenever the file is fragmented. But this calculation is confounded by other processes also executing I/O to the disk at the same time. This overhead is part of the reason why swap and other intensive I/O files should not be put on the same disk.

One mechanism for speeding up disk I/O is to stripe disks. Disk striping allows data from a particular file to be spread over several disks. Striping allows reads and writes to be performed in parallel across the disks without requiring any application changes. This can speed up disk I/O quite effectively. However, be aware that the seek and rotational overhead previously listed still applies, and if you are making many small random reads, there may be no performance gain from striping disks.

Finally, note again that using remote disks adversely affects I/O performance. You should not be using remote disks mounted from the network with any I/O-intensive operations if you need good performance.

Clustering Files

Reading many files sequentially is faster if the files are clustered together on the disk, allowing the disk-head reader to flow from one file to the next. This clustering is best done in conjunction with defragmenting the disks. The overhead in finding the location of a file on the disk (detailed in the previous section) is also minimized for sequential reads if the files are clustered.

If you cannot specify clustering files at the disk level, you can still provide similar functionality by putting all the files together into one large file (as is done with the ZIP file systems). This is fine if all the files are read-only files or if there is just one file that is writable (you place that at the end). However, when there is more than one writable file, you need to manage the location of the internal files in your system as one or more grow. This becomes a problem and is not usually worth the effort. (If the files have a known bounded size, you can pad the files internally, thus regaining the single file efficiency.)

Cached File Systems (RAM Disks, tmpfs, cachefs)

Most OS provide the ability to map a file system into the system memory . This ability can speed up reads and writes to certain files in which you control your target environment. Typically, this technique has been used to speed up the reading and writing of temporary files. For example, some compilers (of languages in general, not specifically Java) generate many temporary files during compilation. If these files are created and written directly to the system memory, the speed of compilation is greatly increased. Similarly, if you have the a set of external files that are needed by your application, it is possible to map these directly into the system memory, thus allowing their reads and writes to be speeded up greatly.

But note that these types of file systems are not persistent. In the same way the system memory of the machine gets cleared when it is rebooted, so these file systems are removed on reboot. If the system crashes, anything in a memory-mapped file system is lost. For this reason, these types of file systems are usually suitable only for temporary files or read-only versions of disk-based files (such as mapping a CD-ROM into a memory-resident file system).

Remember that you do not have the same degree of fine control over these file systems that you have over your application. A memory-mapped file system does not use memory resources as efficiently as working directly from your application. If you have direct control over the files you are reading and writing, it is usually better to optimize this within your application rather than outside it. A memory-mapped file system takes space directly from system memory. You should consider whether it would be better to let your application grow in memory instead of letting the file system take up that system memory. For multi-user applications, it is usually more efficient for the system to map shared files directly into memory, as a particular fule then takes up just one memory location rather than duplicate in each process. Note that from SDK 1.4, memory-mapped files are directly supported from the java.nio package. Memory-mapped files are slightly different from memory-mapped file systems. A memory-mapped file uses system resources to read the file into system memory, and that data can then be accessed form Java through the appropriate java.nio buffer. A memory-mapped file system does not require the java.nio package and, as far as Java is concerned, files in that file system are simply files like any others. The OS transparently handles the memory mapping.

The creation of memory-mapped file systems is completely system-dependent, and there is no guarantee that it is available on any particular system (though most modern OS do support this feature). On UNIX system, the administrator needs to look at the documentation of the mount command and its subsections on cachefs and tmpfs. Under Windows, you should find details by looking at the documentation on how to setup a RAM disk, a portion of memory mapped to a logical disk drive.

In a similar way, there are products available that pre-cache shared libraries (DLL) and even executables in memory. This usually means only that an application starts quicker or loads the quicker, and so may not be much help in speeding up a running system.

But you can apply the technique of memory-mapping file systems directly and quite usefully for applications in which processes are frequently started. Copy the Java distribution and all class files (all JDK, application, and third-party class files) onto a memory-mapped file system and ensure that all executions and classloads take place from the file system. Since everything (executables, DLLs, class files, resources, etc.) is already in memory, the startup time is much faster. Because only the startup (and class loading) time is affected, this technique gives only a small boost to applications that are not frequently starting processes, but can be usefully applied if startup time is a problem.

Disk Fragmentation

When files are stored on disk, the bytes in the files are note necessarily stored contiguously: their storage depends on file size and contiguous space available on the disk. This non-contiguous storage is called fragmentation. Any particular file may have some chunks in one place, and a pointer to the next chunk that may be quite a distance away on the disk.

Hard disks tend to get fragmented over time. This fragmentation delays both reads from files (including loading applications into computer memory on startup) and writes to files. This delay occurs because the disk header must wind on to the next chunk with each fragmentation, and this takes time.

For optimum performance on any system, it is a good idea to periodically defragment the disk. This reunites files that have been split up so that disk heads do not spend so much time searching for data once the file-header locations have been identified, thus speeding up data access. Defragmenting may not be effective on all systems, however.

Disk Sweet Spots

Most disks have a location from which data is transferred faster than from other locations. Usually, the closer the data is to the outside edge of the disk, the faster it can be read from the disk. Most hard disks rotate at constant angular speed. This means that the linear speed of the disk under a point is faster the farther away the point is from the center of the disk. Thus, data at the edge of the disk can be read from (and written to) at the faster possible rate commensurate with the maximum density of data storable on disk.

This location with faster transfer rates usually termed the disk sweet spot. Some
(Commercial) utilities provide mapped access to the underlying disk and allow you to reorganize files to optimize access. On most server systems, the administrator has control over how logical partitions of the disk apply to the physical layout, and how to position files to the disk sweet spots. Experts for high-performance database system sometimes try to position the index tables of the database as close as possible to the disk sweet spot. These tables consist of relatively small amounts of data that affect the performance of the system in a disproportionately large way, so that any speed improvement in manipulating these tables is significant.

Note that some of the latest OS are beginning to include “awareness” of disk sweet spots, and attempt to move executables to sweet spots when defragmenting the disk. You may need to ensure that the defragmentation procedure does not disrupt your own use of the disk sweet spot.

The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java.

I would also recommend visiting the site author by Jack himself and a couple of his mates. It is a very resourceful site for Java performance-related information. Click here to access it.

Related Topic

Basics: CPU Bottlenecks

2007-07-01T11:32:00.000+08:00

Java provides a virtual machine runtime system that is just that: an abstraction of a CPU that runs in software. (Note that this chapter is taken from the "Java Performance Tuning" written by Jack Shirazi and therefore alot of discussions circled around Java technologies.) These virtual machines run on a real CPU, and in this section the book discuss the performance characteristics of those real CPUs.

CPU Load

The CPU and many other parts of the system can be monitored using system-level utilities. On Windows, the task manager and performance monitor can be used for monitoring. On UNIX, a performance monitor (such as perfmeter) is usually available, as well as utilities such as vmstat. Two aspects of the CPU are worth watching as primary performance points. These are the CPU utilization (usually expressed in percentage terms) and the run-able queue of processes and threads (often called the load or the task queue). The first indictor is simply the percentage of the CPU (Or CPUs) being used by all the various threads. If this is up to 100% for significant periods of time, you may have a problem. On the other hand, if it isn’t, the CPU is under-utilized, but that is usually preferable. Low CPU usage can indicate that your application may be blocked for significant periods on disk or network I/O. High CPU usage can indicate thrashing (lack of RAM) or CPU contention (indicating that you need to rune the code and reduce the number of instructions being processed to reduce the impact on the CPU).

A reasonable target is 75% CPU utilization (which from what I read from different authors varies from 75% till 85%). This means that the system is being worked toward its optimum, but that you have left some slacks for spikes due to other system or application requirements. However, note that if more than 50% of the CPU is used by system processes (i.e. administrative and IS process), your CPU is probably under-powered. This can be identified by looking at the load of the system over some period when you are not running any applications.

The second performance indicator, the run-able queue, indicates the average number of processes or threads waiting to be scheduled for the CPU by the OS. They are run-able processes, but the CPY has no time to run them and is keeping them waiting for some significant amount of time. As soon as the run queue goes above zero, the system may display contention for resources, nut there usually some value above zero that still gives acceptable performance for any particular system. You need to determine what that value is in order to use this statistics as a useful warning indicator. A simplistic way to do this is to create a short program that repeatedly does some simple activity. You can then time each run of that activity. You can run copies of this process one after the other so that more and ore copies are simultaneously running. Keep increasing the number of copies being run until the run queue starts increasing. By watching the times recorded for the activity, you can graph that time against the run queue. This should give you some indication of when the run-able queue becomes too large for useful responses on your system, administrator if the threshold is exceeded. A guideline by Adrian Cockcroft is that performance starts to degrade if the run queue grows bigger than four times the number of CPUs.

If you can upgrade the CPU of the target environment, doubling the CPU speed is usually better than doubling the number of CPUs. And remember that parallelism in an application doesn’t necessarily need multiple CPUs. If I/O is significant, the CPU will have plenty of time for many threads.

Process Priorities

The OS also has the ability to prioritize the processes in terms of providing CPU time by allocating process priority levels. CPU priorities provide a way to throttle high-demand CPU processes, thus giving other processes a greater share of the CPU. If there are other processes that need to run on the same machine but it doesn’t matter if they were run slowly, you can give your application processes a (much) higher priority than those other processes, thus allowing your application the lion’s share of CPU time on a congested system. This is worth keeping in mind if your application consists of multiple processes, you should also consider the possibility of giving your various processes different levels of priority.

Being tempted to adjust the priority levels of processes, however, is often a sign that the CPU is underpowered for the tasks you have given it.

The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java.

I would also recommend visiting the site author by Jack himself and a couple of his mates. It is a very resourceful site for Java performance-related information. Click here to access it.

Related Topic

Basics: Underlying Operating System and Network Improvements

2007-07-01T11:11:00.000+08:00

If you control the OS and hardware where the application will be deployed, there are a number of changes you can make to improve performance. Some changes are generic and affect most applications, while some are application-specific. This article applies to most server systems running Java application, including servlets, where you usually specify (or have specified to you) the underlying system, and where have some control over tuning the system. Client and standalone Java programs are likely to benefit from this chapter only if you have some degree of control over the target system, but some tips in the chapter apply to all Java programs.

It is usually best to target the OS and hardware as a last tuning choice. Tuning the application itself generally provides far more significant speedups than tuning the systems on which the application is running. Application tuning also tends to be easier (though buying more powerful hardware components is easier still and a valid choice for tuning). However, application and system tuning are actually complementary activities, so you can get speedups from tuning both the system and the application if you have the skills and resources. Here are some general tips for tuning systems:

Constantly monitor the entire system with any monitoring tools available and keep records. This allows you to get a background usage pattern and also lets you compare the current situation with situations previously considered stable.

You should run offline work during off-hours only. This ensure that there is no extra load on the system when the users are executing online tasks and enhance performance of both online and offline activities.

If you need to run extra tasks during the day, try to slot them into times with low user activity. Office activity usually peaks at 9am and 2:30pm and has a load between noon and 1pm or at shift changeovers. You should be able to determine the user-activity cycles appropriate to your system by examining the results of normal monitoring. The reduced conflict for system resources during periods of low activity improves performance.

You should specify timeouts for all process under the control of your application (and others on the system, if possible) and terminate processes that have passed their timeout value.

Apply any partitioning available from the system to allocate determinate resources to your application. For example, you can specify disk partitions, memory segments, and even CPUs to be allocated to particular process.

As the entire chapter is lengthy, I've split them into various sections namely the links that followed. Click on the links to access the article.

The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java. A simplified version (which is the summary of the chapter can be found here [Comming soon]).

I would also recommend visiting the site author by Jack himself and a couple of his mates. It is a very resourceful site for Java performance-related information. Click here to access it.

Related Topic

Transaction Response Time

Basics: Max. Running Vusers in Analysis is different from Vuser Quantity in Controller?

2007-06-29T09:09:00.000+08:00

There are couple of times when an individual starts to pick up LoadRunner ask this question. I've written the explanation here for the convenience of the newbies which I had previously responded in the Yahoo! group discussion.

Max. Running Vuser

It defines at any point of point of time, the maximum number of vusers running together concurrently (in Run state). This is the "state" or usually the requirement of a load test to reach "X" number of concurrent users. If a load test were to required to run 100 concurrent users, then the Max. Running Vuser must be 100. This is different from Vuser Quantity explained in the following.

Vuser Quantity

In the Controller, the Vuser Quantity is the total number of participating in the load test but it is different from Max. Running Vuser which is explained above. To view the total number of Vuser participating the load test (Vuser Quantity), open the Vuser Summary graph. Please take note that the Summary Report in the Analysis session displays the maximum number of vusers running in the scenario and not the total vuser particiapting.

Related Topics

Protocols: Java Error - java.lang.NoClassDefFoundError

2007-06-28T17:42:00.000+08:00

I had a question from a visitor with regards to a Java error. The error message is as followed.

java.lang.NoClassDefFoundError: jhook/HookMetaData
at java.rmi.Naming.lookup(Naming.java)
at pie.engine.atp.ClientRMI.getRemoteModuleImpl(ClientRMI.java:256)
at pie.engine.atp.ClientATPRemoteEx.getRemoteModule(ClientATPRemoteEx.java:89)
at pie.engine.atp.ClientATPRemoteEx.getRemoteModule(ClientATPRemoteEx.java:64)
at pie.engine.atp.VirtualRemoteServer.setupPingRemote(VirtualRemoteServer.java:102)
at pie.engine.atp.VirtualRemoteServer.pingServer(VirtualRemoteServer.java:166)
at pie.engine.atp.CacheTimer$1.run(CacheTimer.java:47)
at java.util.TimerThread.mainLoop(Timer.java:432)
at java.util.TimerThread.run(Timer.java:382)

I did not have straight answer for him initially but, he was able to resolve the problem by himself with the following command script (inclusion of the Xbootclasspath).

-Xbootclasspath/p:"D:\Program Files\LoadRunner\classes;D:\Program Files\LoadRunner\classes\srv"

Once that was added, the script worked for him! Well hope this is useful to anyone that is encountering the same problem.

Related Topic

Working with CORBA-Java
Working with RMI-Java
Working with Java - Introduction

Monitors: The Other Way? II

2007-06-25T14:58:00.000+08:00

The organisation that I worked in have frequent port clearance to perform prior the load test and this is often a troublesome matter to do. As you know that for monitoring to be successful, it requires certain ports to be opened as described in the article titled, "Monitors: How does the monitoring work?". So for this requirement, port-clearance becomes part and parcel of the performance tester job (or for me in particular). This kind of situation exists for some other organisations too.

With that, I introduce you to monitor in an alternative way by monitoring locally on the machine and logging the data. The monitoring fundamental for LoadRunner is almost the same for all monitors, as such, you can collect the performance on the machine locally then port it to Analysis to be imported. Analysis provide an import feature that allows external monitor to be ported into the results of the load test.

The import feature is straight forward in the usage. More information (a short one) can be found in the Analysis User Guide that comes along with every installation of LoadRunner. I would advise the data to be stored into a .csv file for easier import as Analysis can extract the data easily.

An example of collecting data on the local machine for Windows can be found in the article, "How to create a log using System Monitor in Windows". Another article titled, "How to create a Performance Monitor log in Windows NT". I will come up an example for UNIX in my future post.

Somethings to note, the timing of the graphs may deviate more than often. What I suggest is try to synchronize the timing of the local machine to the load test results for an accurate reading whenever possible.

Related Topics

Monitors: The Other Way?

General: Maximum amount of Vusers generated by a single load generator

2007-06-19T22:34:00.000+08:00

How many vusers can a Load Generator generate?

There is a straight forward answer in this KB article, KB 11808, "How many Vusers can run on a given computer" from Mercury/HP. Also, you can refer to the following article, KB 3629, "Recommended hardware and OS tuning for optimal Vuser performance" to get information of recommended hardware for vuser performance. As per metioned in the articles, the question of maximum load relates closely to the CPU and Memory on a given hardware.

Before we dwell deeper into the technical details, it is best to ensure that common blunders are removed, such as, (1) the Controller have sufficient license to run the amount of vusers stated in the Scenario. (2) Furthermore, ensure that there are no limits enforced on the Load Generators with are configured in the Scenario Settings.

To start with, we must understand what happens during a scenarion execution. Refer to General: Scenario Execution for an understanding of the scenarion execution. Furthermore, when vusers are executed in a scenario, the vusers are actually running in memory instead of the disk. That is why memory is an important factor to consider.

Also, the article, KB 10291, "Vuser Memory requirements - memory footprints" provides a list of memory footprint of different types of protocols. This is only a measurement of a simple script of its kind but it can be used to estimate roughly the amount of load generated on a given hardware specification.

For Citrix, SAP GUI and RTE, it is closely related to GDI resources. GDI resources is a hard-coded value in the OS and cannot be changed. For more information, you can refer to the Microsoft article on GDI Objects. To monitor GDI resource, you can refer to KB 30684, "How to monitor the GDI resources used".

If you like to maximize the amount of load that can be generated, you can refer to the following article, KB 3645, "How to increase the number of Vusers you can run on a Windows Machine" by Mercury/HP. However, this is not applicable to GDI resources-related protocols.

WIth all suggestions above, the best bet to determine the amount of load generated, is to monitor the Load Generators itself (instead of the machines under load in a load test).

Take note that the above Mercury/HP links required a valid login to their support site. Please apply an account with them first before you click on them.

Related Topics

Basics: Transaction Response TIme

2007-06-18T22:52:00.000+08:00

What is Transaction Response Time?

Transaction Response Time represents the time taken for the application to complete a defined transaction or business process.

Why is important to measure Transaction Response Time?

The objective of a performance test is to ensure that the application is working perfectly under load. However, the definition of “perfectly” under load may vary with different systems.
By defining an initial acceptable response time, we can benchmark the application if it is performing as anticipated.

The importance of Transaction Response Time is that it gives the project team/ application team an idea of how the application is performing in the measurement of time. With this information, they can relate to the users/customers on the expected time when processing request or understanding how their application performed.

What does Transaction Response Time encompass?

The Transaction Response Time encompasses the time taken for the request made to the web server, there after being process by the Web Server and sent to the Application Server. Which in most instances will make a request to the Database Server. All this will then be repeated again backward from the Database Server, Application Server, Web Server and back to the user. Take note that the time taken for the request or data in the network transmission is also factored in.

To simplify, the Transaction Response Time comprises of the following:

Processing time on Web Server
Processing time on Application Server
Processing time on Database Server
Network latency between the servers, and the client

The following diagram illustrates Transaction Response Time.

Figure 1 Transaction Response Time = (t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9) X 2

Note:
Factoring the time taken for the data to return to the client.

How do we measure?

Measuring of the Transaction Response Time begins when the defined transaction makes a request to the application. From here, till the transaction completes before proceeding with the next subsequent request (in terms of transaction), the time is been measured and will stop when the transaction completes.

Differences with Hits Per Seconds

Hits per Seconds measures the number of “hits” made to a web server. These “hits” could be a request made to the web server for data or graphics. However, this counter does not represent well to users on how well their applications is performing as it measures the number of times the web server is being accessed.

How can we use Transaction Response Time to analyze performance issue?

Transaction Response Time allows us to identify abnormalities when performance issues surface. This will be represented as slow response of the transaction, which differs significantly (or slightly) from the average of the Transaction Response Time.

With this, we can further drill down by correlation using other measurements such as the number of virtual users that is accessing the application at the point of time and the system-related metrics (e.g. CPU Utilization) to identify the root cause.

Bringing all the data that have been collected during the load test, we can correlate the measurements to find trends and bottlenecks between the response time, the amount of load that was generated and the payload of all the components of the application.

How is it beneficial to the Project Team?

Using Transaction Response Time, Project Team can better relate to their users using transactions as a form of language protocol that their users can comprehend. Users will be able to know that transactions (or business processes) are performing at an acceptable level in terms of time.

Users may be unable to understand the meaning of CPU utilization or Memory usage and thus using a common language of time is ideal to convey performance-related issues.

Related Topics:

What are SiteScope components?

Products: SiteScope

2007-06-17T15:26:00.000+08:00

SiteScope is another Mercury/HP product that provides agentless monitoring to the users. Agentless monitoring in this context means that there is no need to have additional installation of components to the target servers to perform monitoring.

Similar to the LoadRunner Controller, SiteScope requires the designated server to be able to perform monitoring of its own which SiteScope will be able to draw data from it. Depending on the user, they have a choice of using SiteScope to perform the monitoring or use the Controller.

Below is a list of links written by Charles Laudia that provides SiteScope-related information.

What is SiteScope?

How does SiteScope work?

How does the licensing work in SiteScope?

SiteScope for Windows QuickStart

Related Topics

Content Page - Products

Working with Web (HTTP/HTML) V: File Download/Export

2007-06-08T20:58:00.000+08:00

Frequently, I've observed that there are always questions relating to file download functionalities or file export not being recorded by Vugen for Web (HTTP/HTML) protocol in Web applications, such as clicking a link and the file will be downloaded.

In this article, I will describe the possibilities of the problem based on my experiences. We will elaborate using a Winword file.

When you click on the link or button to access the file, the file gets loaded in the browser. From here, the business process requires you to click on a link to export it or save it to the local machine.

There are two possibilities that will happen. The Winword is either saved over the network or at the OS level.

In the context of network level, the client makes a request to the server (as per norm) to download the file which translates to network traffic going to and fro between the client and server. This will be recorded.

In the context of OS level, the client issues a OS command to "save-as" the file from a temporary location (e.g. Temporary Internet Folder) to a defined location. Why is that so? Sometimes, the file is actually saved in the temporary folder in the first file download request which is viewed in the browser and the subsequent "export" command is actually an OS command to retrieve the file from the temporary location. This usually happens for Web applications that requires another application, in this example (Winword Viewer) to be launched prior the file to be viewed in the browser.

Therefore, in the two situations, you may want to know when the file is actually downloaded. And for this, I would recommend turing on the Full Extended Log to dig further. Usually for file downloads, the data should be represented in mixed chunk of characters (or what we defined them as binaries) in the Full Extended Log.

However, before you start with digging the logs, I would also recommend you to look at this previous article, "Protocols: Working with Web (HTTP/HTML)" that have recording solutions for Web (HTTP/HTML) protocols. This will at least scope the problem to application related or configuration setup in Vugen.

Related Topics

Protocols: Working with Web (HTTP/HTML)

General: Virtualization with LoadRunner

2007-06-06T17:14:00.000+08:00

There was the question posted in the Yahoo discussion group on the topic of virtualization with LoadRunner. That is to use virtualization tools such as VmWare and its support and compatibility with LoadRunner.

James Pulley who often posts in the group came together a list of recommendations with regards to his experience in such a topic here. The recommendations consist of what could be installed on the virtual machines. I personally appreciated that he put in these information.

I believed that the recommendations will be very useful for individuals or organisations looking for some answers with the proliferation of the virtualization tools and therefore extracted the content of the discussion and publish here.

The recommendation by James were in order of the least influenced to the most influenced by virtualization.

Analysis

The core issue associated with virtualization is the state of the virtualized system clock which is inconsistent in operation, so much so that time critical events do not illustrate reliable behavior Mercury Analysis. This is a very easily virtualized tool. There are no time critical components associated with the analysis engine, since all of the data has been collected by other hosts during the execution of the performance test. This virtualized instance works best when the data store for the testr esults is in a Microsoft SQL Server database as opposed to the default datastore of a Microsoft Access formatted data file. With the change inl icensing with 8.1, this item can be promiscuous in its distribution and is no longer limited to just a single copy with a controller purchase.

LoadRunner Virtual User Generator

As the workbench for the script development, there are very few time critical events associated with the development and the testing of scripts. Virtualized forms of VUGEN work best on platforms which have hardwarevirtual machine extensions, such as the latest versions of the Intel CoreDuo processor and their AMD equivalents. Hyperthreaded machines seem to show in consistent performance as well - There is a knowledge base article on this hyperthreaded issue in the Mercury knowledge base. Timing may be a little off in the log due to the clock issue noted above, but other than thisissue, the VUGEN operates very well in a virtualized form. As with the Analysis engine, with the change in licensing with 8.1, this item can be promiscuous in its distribution and is no longer limited to just a singlecopy with a controller purchase.

LoadRunner Controller

As we move into the controller timing becomes a more critical item. The controller itself can be virtualized without an issue, but because thetiming record information collected is as an offset to the beginning of thetest, an inconsistent clock on the controller can become an issue formonitor data with offset timestamps may not reflect the actual system clock.

Load Generator/Injector

This is a layer which suffers from the most influence from the inconsistent clock issue for time critical events within a virtual machine and of all ofthe layers, this is the one which should be virtualized the least. The best test design would dictate that all of your generators have an identical hardware and software profile, a very difficult item to achieve invirtualized form which suffers from a virtualized CPU and a virtualized system clock.This time critical items is not something which would just impact LoadRunner. It also impacts other time critical processes, such as inability to run some functions for conferencing in virtualized PBXs.

Related Topics:

References extracted from the discussion topic:

http://www.vmware.com/pdf/vmware_timekeeping.pdf

http://kb.vmware.com/selfservice/viewContent.do?externalId=1420

http://msmvps.com/blogs/rexiology/archive/2006/12/21/clock-in-a-linux-guest-runs-more-slowly-or-quickly-than-real-time.aspx

http://www.mydigitallife.info/2006/04/12/freebsd-60-on-vmware-server-time-and-clock-slow-down/

Protocols: Working with CORBA-Java

2007-06-01T14:52:00.000+08:00

The blog by Stuart Moncrieff has great resources on load testing and particularly one of them is CORBA-Java. It comes really handy with a downloadable document at the end of the article and when Java-related protocol resources are hard to find.

The article "Tips for Load Testing CORBA Applications with LoadRunner" covers recording, replaying, workarounds and scripting. Click on the link here to be redirected.

Related Topics

Products: Diagnostics

2007-05-31T12:17:00.000+08:00

The Mercury Diagnostics Software is a standalone software of its own that is functional by itself. Its main functionality is to profile the J2EE, .NET or ERP/CRM application and allowing drill down of the modules to understand the actual problem cause (depending on what you’ve purchased and installed).

If you would like to hear from the sales, click here.

The discussion here will be specified to performance testing (LoadRunner/Performance Center) and J2EE.

The Diagnostics setup comprises of the Probe/Profiler and the Server (Commander). Note that Probe and Profiler will be use interchangeably in this article. Previous version of Diagnostics, 4.2 and earlier have the Probe/Mediator/Commander however that was been incorporated into just the Probe and Commander. Note as of this writing, the latest Diagnostics is 6.5

The Profiler/Probe is also able to function on its own by collecting data of the instrumented application server. This can be displayed via the browser via port 35000 by default or the Java Profiler application. Built together with the Diagnostics Server, the data collected are sent to the server for reporting and display.

In order to make them work, the application server must be instrumented for the modules to be profiled. This is done by adding an additional code in the startup script for the application server. Instructions are detailed for each type of application server (e.g. BEA WebLogic, IBM WebSphere, etc). A further description of how they work is detailed in the vendor website here.

So with Diagnostics, how does it complement LoadRunner? LoadRunner does not report specified/drill down to module level, or provide information of problematic modules (that are causing the bottleneck). It only reports system bottleneck in a bigger picture, such as CPU, Memory or Disk usage. With Diagnostics, you can profile modules that are causing the bottlenecks and determine the next course of action.

Significant limitations to note, Diagnostics can report only modules/calls that are made on the Application Server. Therefore, Store Procedure Calls on the Database Servers will not be reported during the profiling. However, note that this is not the only limitation listed.

Prior to the instrumentation, enquire the support level for the probe with the Application Server with Mercury/HP support. They will be able to advise if the instrumentation is supported or not supported. But having said that, the instrumentation and support level is quite dependant on the type of JDK that the Application Server is running on (point to note).

Also, most of the time, when the Probes are not reporting any data, it would most likely be due to the application script not being instrumented properly. In such events, review the startup script of the application servers again.

To start been familiarizing the Diagnostics Software, it will be good to spend a day or two to download a J2EE sample application server and instrument it. BEA WebLogic is a good source that provide sample application server.

The licensing is bounded by the number of probes that can be installed and the number of Diagnostics Server that is been implemented. This license should not be confused with the one associated with LoadRunner/PC. In order for Diagnostics to work with LoadRunner/PC, a separate license called Diagnostics Module License is required. This will allow the reporting of the Diagnostic server to be incorporated into the load test result.

When working with LoadRunner/PC, always ensure that the Diagnostics setup is fully functional before adding the Diagnostics Module into the LoadRunner monitoring. This will at least scope the problem cause when troubleshooting.

Lastly, in previous version of LoadRunner, Java Performance Monitoring was offered to the user. However the installation as described in the documentation is inconsistent. Moving forward, J2EE Diagnostics is to “replace” its processor, Java Performance Monitoring.

Related Topics

Content Page - Products

General: Vugen/Controller Crash or Abnormal Behavior

2007-05-28T17:56:00.001+08:00

In the event that Vugen, Mercury/HP has provided a document that assists the user in troubleshooting the problem. The document, KB 31731 provided is rather old but still relevant.

Download the document and follow the steps sequentially not to miss any of it. Do take note on the following three which you can start with which usually resolves the problem.

1. Login as administrator
2. Run the batch file
3. Reinitializing the ini file

The definition of administrator here is it is a default administrator on the OS without any changes to its roles or privileges.

Similarly, for crashing or abnormal behavior of Controller, you can refer to KB 12532. Take note that for both KB, you will need a valid login to the support website to access them.

On top of the list of solutions provided by Mercury/HP, you may want to verify the following too.

1. Ensure that there are no other Mercury/HP product (especially QTP or WinRunner). It is advisable to install only LoadRunner components on the machine.

2. Ensure that that DEP (Data Execution Prevention) is disabled for Vugen and the client program in Windows XP SP2. More details can be found in KB 41489.

3. When working with Web (HTTP/HTML) protocol, it may be the application that is causing the problem (crash/hang) rather than Vugen. Under this circumstances, refer to the article, "Working with Web(HTTP/HTML)" to troubleshoot the problem.

If all else fails, the best suggestion is to get a “clean” machine and login with default local administrator and perform a reinstallation. The definition of “clean” means that the machine OS is not built on a clone image.

Related Topics

Scripts: Replay Failure – Use Full Extended Log!

2007-05-22T20:20:00.000+08:00

Usually after recording, you would like to replay the script to ensure that it is workable. More than ever, if it is Web Application (or other application) that have a login and DB connection mechanism, it will fail.

The above can be caused by various problems that it will be difficult to list them all out in this article. But for a start, the most common for Web Applications are:

1. Session ID
2. Authentication (Basic, NTLM or Digest)
3. Certificates

When the errors happened, amateur users of LoadRunner usually do not know how to follow up.

My recommendation for newbie when such things happened is to turn on the Full Extended Log and trace the possible problem areas. The Full Extended Log provides information that you can trace what is happening to the data being sent by the client and responded from the server. In this way, you will be able to determine if the correct request (in web application context) is sent to the server and if a valid respond is been sent back from the server.

Trace carefully every request made and the response. They will return header information, actual data in text or binary, and many other more information. A good knowledge of HTTP response will be very useful when working in Web (HTTP/HTML) protocols.

My other recommendation is not too over rely on the Runtime Viewer. The main problem is that it may not support all forms of HTTP/HTML for display and thus unable to provide much assistance to you. More details of the limitations of Runtime Viewer can be found in “What are the limitations of the Run-Time Viewer” from Mercury/HP support website.

The Full Extended Log must be enabled with the following three options to be really useful:

1. Parameter substitution
2. Data returned by server
3. Advance data trace

Take note that Full Extended Log is applicable to almost all protocols and it's a useful tool to diagnose problem during script replay failure.

However, take note that Full Extended Log will generate a huge amount of log file that may be resource intensive for a big script. And when it is been ported over to a scenario execution, it will manifest to become a bigger resource/performance issue.

Therefore, prior to a scenario execution, remember to turn off the Full Extended Log unless you have a purpose for it.

Related Topics

Content Page - Scripts

Message: Change of WebSite Template

2007-05-21T21:39:00.000+08:00

Hi Visitor,

If you are a returning visitor, you might have noticed that the layout of this blog have changed. Well not really a perfect one, but sufficient for you to navigate (with the search box and menu on the top). Feel free to comment or suggest a good template to contribute effort of improving this website with content and user interface.

Regards
Hs

Working with Java – Introduction

2007-05-20T16:34:00.001+08:00

LoadRunner provides the capability of load testing Java-related applications. In this article, we will provide a brief introduction to working with Java related applications in LoadRunner.

[1]
Before we start choosing Java-related protocols to perform recording, you will need to understand the protocol that the Java application is using to communicate with the server. It maybe the case where the Java application is communicating via HTTP rather than RMI-Java, etc… Refer to the concept in a previously written article, "Protocols: What Protocol to use?". Take note that it is the fundamental of using LoadRunner correctly.

If you have identified that the application is communicating with a Java related protocol, you can safely proceed to choose the relevant Java protocol.

[2]
For Java related protocol, the recording works in the same way as .NET and COM/DCOM protocol. All of them share the same characteristics but I will specify to just Java and only Java recording. Pure Java scripting will be excluded in this article.

Unlike .NET and COM/DCOM, Vugen provides an interface to manage and filter the classes and methods via the GUI which is very convenient to the user. Hopefully, Mercury/HP may consider that in future.

[3]
Java recording is not as straight-forward as popular (simple) protocols such as Web (HTTP/HTML) and the following lists out the items to take note when working with Java.

Classes and Methods

1. You will need the knowledge of the classes and methods prior recording; and usually this information is held by the developers.

2. The need to know the classes and recording is crucial because, LoadRunner (Vugen) implements a hooking mechanism to hook the traffic on the classes. Not knowing or minimal knowledge will definitely delay the recording.

3. LoadRunner does have a default hook for Java but if you have additional classes that you need to record. The knowledge of the classes will come important here.

4. Also, by defining all classes to hook, is a wasteful effort as you maybe recording classes’ activity that is not useful to the load test. They may comprise of GUI clicks or client activities.

JDK and JRE

1. You will need to know the JDK of the application. By default, Sun Microsystems JDK (SDK) works with LoadRunner.

2. You can check with Mercury/HP on the support for other JDKs (IBM, etc.). However, usually they are not supported.

3. The least supported JDK is JDK 1.3 for LoadRunner 8.1. Take note that the recording is still very dependant on the JDK version installed. That is, if your application is JDK 1.6, you will have to install that version on the recording machine.

4. JRE is insufficient for the recording to be successful. JDK is required to be installed.

Recording-specified

1. Vugen implements an API that hooks into the selected classes and record the traffic. Therefore, as pointed earlier, knowing the correct classes to hook is crucial.

2. GUI classes or client-side activities maybe recorded via the hook. However, this is not recommended as it does not contribute to the load generation towards the server.

3. For recording specifies, I’ve written another article that specifies the steps and the implications of each step in this article, "Working with RMI-Java" .

Scenario Execution

1. Prior the scenario execution in the Controller, ensure that JDK have be installed on the Load Generators. The scripts sent (refer to the article, "General: Scenario Execution" for more information) are Java scripts, and require JVM to run.

[4]

Documentations

If you like to know more information from the official provider, you can refer to the Vugen User Guide that is provided in each installation of LoadRunner in PDF format. Specifically the following chapters:

1. All chapters in Part II: Working with Java Language Protocols
2. Chapter 18: Developing Corba-Java Vuser Script
3. Chapter 19: Developing RMI-Java Vuser Scripts.

Sample

Also, if you like to get a feel of working with Java protocol, I would recommend using the sample example provided with every installation of LoadRunner. It will at least get you understand what will happened and the expected output when recording Java language applications.

[5]
In summary, we’ve covered the workings of Java language protocol in LoadRunner, things to watch out and additional resources to pertaining to the topic. It should be sufficient for you to work with Java language protocols at minimal difficulty.

Related Topic:

Content Page – Protocols

Working with RMI-Java

Protocols: Working with Winsock

2007-05-19T16:15:00.000+08:00

LoadTester have another article that discuss interestingly on the complex protocol Winsock. Click here to download the article.

The article give a brief run through of (1) what is Winsock protocol, (2) the workings of Winsock, (3) how is it being used under what situations and (4) some exercises.

Although the article provides a brief description of the Winsock protocol, it at least helps to clarify some questions that Mercury/HP left out in their knowledge base and documentations.

Related Topics:

Content Page – Protocols

Monitors: MS SQL Server

2007-05-17T10:17:00.000+08:00

LoadTester also provides another white paper that discussed about the performance counters you can utilize for monitoring SQL Server. I've extracted the portion of it and paste it here which will be useful first-hand for anyone that requires to monitor SQL Server. The list they provided however, is a recommendation list and you may include more counters based on your requirements.

Cache Hit Ratio (SQLServer: Cache Manager object): This is the % of time that a record was found in cache. The recommend cache hit ratio is 90% or more. For typical OLTP systems, the recommended value is 99%. This can be achieved by adding more RAM to the system and a value of 90% is considered a low number.

In OLAP systems, this % can be much less due to the nature of how OLAP systems work. Hence, the recommended ratio of 90%.

User Connections (SQLServer: General Statistics object): this is the value of number of users connected to this database. Note that this value is the total number of user connections and not the total number of users. By default, SQL Server is configured for 255 user connections. If this value is exceeded, then increase the value of "Maximum Worker Threads" to a number higher then 255. As a rule, this value should be higher then the number of user connections. You can also set the maximum concurrent user connections to ‘0’. Which means that the number of connections is limited only by the SQL Server maximum.

Transactions/sec (SQLServer: Databases object): The number of transactions started for the database. These transactions come in the form of requests from client machines that are services by the database. Having too many concurrent transactions can cause locking problems and incur additional overhead.

Data File(s) Size (KB) (SQLServer: Database object): The cumulative size of the data fiels that reside on a disk array. This counter is useful in tracking disk usage growth for capacity planning studies.

Percent Log Used (SQLServer: Databases object): The percentage of the log that is used. This counter is helpful in tracking log growth in capacity planning studies.

The actual document can be found here.

Related Topics:

Protocols: Working with Citrix II - Best Practices

2007-05-16T17:23:00.001+08:00

I’ve recently come across a white paper regarding load-testing Citrix, provided by LoadTester Incorporated. I believed that it is very useful for those that are having problems with the working with Citrix as it covers quite a few aspects in this area.

The white paper covers the following best practices for the following.

Display Settings
Keyboard Entry versus the Mouse
ICA Files
Randomizing Information
Debugging Vusers in the Controller
Generate Snapshot on Error
Continue on Error
Scripting without the Citrix Agent
Scripting with the Citrix Agent

Click here to access the actual document.

Related Topics:

General: Scenario Execution

2007-05-16T09:56:00.000+08:00

What happens when the scenario is executed? Here, we will discuss the basic overview when the scenario starts execution, during and end.

Prior to the scenario execution, the Load Generators, number of Vusers, the designated monitoring machine, the ramp up timing, so on so forth should be already defined, and we will skip the discussion for it.

When the scenario starts executing, the Controller dispatches the Vusers (scripts) to the designated Load Generator for execution. The Vusers are then executed on the Load Generator with the pre-defined amount on behalf of the Controller. The data generated by the Load Generator prior Collation process (discussed later) is stored in its default temp directory. This can be configured before the execution under the Load Generator setup.

Note that if your load test stretches a (1) long duration, (2) contain additional logs or (3) you foresee a large amount of data to be generated, it is advisable to ensure that you have sufficient space for the temp directory in the Load Generator. This also applies to the Controller temp directory. Either increase the disk size for the temp folder or direct it to a folder that has a larger space.
The temp directory of the Controller and Load Generator serves different purposes. For the Controller, it stores the entire result data of the scenario execution whereas, the Load Generator stores the data of its own during the scenario execution.

During the execution, the real time data of the monitored servers are displayed at the same time displayed in the graphs. This information is logged for further processing when conducting the analyzing.

Point to note; as the scripts are sent to the Load Generator to be executed, it must have the same environmental settings of the recorded scripts (in VuGen). For example, if the script was recorded in RMI-Java, JDK exists on the machine that performed the recording. In the same context, the Load Generator will also require the JDK to exist on it for the script to execute properly.

Another example is an activity of uploading of a file by the script. If the script points to a directory that only exists in the recording machine, the Load Generator will require to create the same directory for it to be executed correctly.

When errors start to occur during a scenario execution, it is best to determine where the error surface in the following questions.

1. Does it happen at the start of the scenario execution?

2. Does it happen at the middle of the scenario execution?

3. If (2), is yes, then how many Vusers were running at that point of time when the errors appeared?

With the above, you can guess if the scenario is actually failing due to the following reasons to list a few.

1. The script is not working properly in the first place or configuration of the script failed.

2. The application is experiencing load during the scenario execution.

3. Network/server issues not pertaining to load have occured during the scenario execution.

When the scenario completes, either (1) the Vusers finished execution, (2) you pressed the "Stop" button or (3) the duration of the load test have reached, the Vusers will gradually exit the scenario. Once all Vusers totally exited, the Controller goes into a Collation process.

In the Collation process, data regarding the execution stored in the temp directory of the Load Generator are been sent back to the Controller to be compiled to one single result data. This is stored in the default folder of the temp directory in Controller, usually in C:\Documents and Settings with the folder name res. The actual file that describes the result is stored as .lrr file extension.

Based on the scenario configuration, the Analysis maybe launched automatically for you to analyze the results directly after the scenario completes. However, you can also choose to launch it manually by double clicking the .lrr file.

With the above, it concludes the entire scenario execution process.

Related Topics:

General: Working with Web (HTTP/HTML) IV: Internet Explorer 7 Support

2007-05-14T21:07:00.000+08:00

IE 7 is supported by LoadRunner 8.1 FP4 with an additional patch. The patch can be downloaded from the Mercury/HP support site here.

Internet Explorer 7 (IE 7) support for LoadRunner 8.1 Feature Pack 4

As of today’s writing, the company has pointed out limitations on the following (taken from the support website)

1. Some versions of JInitiator may cause IE to crash when recording the Oracle NCA protocol on IE 7 platforms.
2. Cannot record the FTP protocol upload or download operations on IE 7 platforms.
3. Does not support tabbed browsing. If you record using VuGen on IE 7 platforms, then Internet Explorer will open with this feature disabled.
4. The current version of Mercury Diagnostics (4.2) does not support IE 7. Diagnostics version 6.5 will support IE 7.

Take note that you need LoadRunner 8.1 installed with FP4 before you can apply the IE 7 patch. If not, the patch installation will not work.

Related Topics

General: Understanding Processor - Interrupt Processing

2007-05-14T17:40:00.000+08:00

Interrupts are subjected to priority. The interrupt priority scheme is hardware-determined, but in the interest of portability it is abstracted by the Windows HAL. During interrupt processing, interrupts from lower-priority interrupts are masked so that they remain pending until the current interrupt processing completes. Following interrupt processing during which interrupts themselves are disabled, the operation system returns to its normal operating mode with the processor reset once more to receive interrupt signals. The processor is once again enabled for interrupts.

Strictly speaking, on an Intel processor, there is a class of interrupts used for switching between the user level and privileged OS code. Although this involves interrupt processing on the Intel microprocessor, we are not referring to that type of interrupts here. Switching privilege levels does not necessarily cause the executing thread to relinquish the processor. However, Windows does classify these OS supervisor call interrupts as context switches.

In Windows, hardware device interrupts are serviced by an interrupt service routine, or ISR, which is a standard device driver function. Device drivers are extensions of the OS tailored to respond to the specific characteristics of the devices they understand and control. The ISR code executes at the interrupt level priority , with interrupts at the same or lower level disabled. An ISR is high priority by definition since it interrupts the regularly scheduled thread and executes until it voluntarily relinquishes the processor (or itself interrupted by a higher-priority interrupt).

The ISR normally signals the device to acknowledge the event, stops the interrupt from occurring, and saves the device status for later processing. It then schedules a deferred procedure call (DPC) to a designated routine that performs the bulk of the device-specified work associated with interrupt processing. DPCs are a special feature of Windows designed to allow the machine to operate enabled for interrupts as much as possible. DPC code executes at a higher priority than other OS privileged modes, but one that does not disable further interrupts from occurring and being serviced.

The DPC mechanism in Windows keeps the machine running in a state enabled for interrupts as much as possible. This architecture is especially useful with Intel PC hardware where many devices connected to a single PCI bus can share the same Interrupt Request Queue (IRQ) level. DPC routines execute from a separate DPC dispatcher queue, which Windows empties before it calls the Scheduler to re-dispatch an ordinary kernel or application thread. A typical function carried out by a DPC routine is to reset the device for the next operation and launch the next request if one is queued. When the DPC completes, a high-priority kernel or device driver thread then marks the I/O function complete. At the end of this chain events, a waiting thread transitions to the ready state, poised to continue processing now that the I/O it requested has completed.

When an interrupt occurs, the thread executing loses control of the processor immediately. When the DPC performing the bulk of the interrupt processing completes, the Windows Scheduler checks the Ready Queue again and dispatches the highest-priority thread. The interrupted thread does not necessarily regain control following interrupt processing because a higher-priority task may be ready to run. This is variously known as preemptive scheduling, preemptive multithreading or preemptive multitasking.

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

General: Understanding Processor - Ready Queue Management

2007-05-14T13:41:00.000+08:00

Think of priority scheduling as the set of rules for ordering the Ready Queue, which is the internal data, structure that points to the threads that are ready to execute. A ready tread (from IE or any other application) transitions directly to the running state, where it executes if no other higher-priority threads are running or waiting. If there is another thread, the Windows Scheduler selects the highest-priority thread in the Ready Queue to run.

Once a thread is running, it executes continuously on the processor until one of the following events occurs:

1. An external interrupt occurs

2. The thread voluntarily relinquishes the processor, usually because it needs to perform I/O

3. The thread involuntarily relinquishes the processor because it incurred page fault, which requires the system to perform I/O on its behalf

4. A maximum uninterrupted execution time limit is reached

Interrupts

An interrupt is a signal from an external device to the processor. Hardware devices raise interrupts to request immediate servicing. An I/O request to a disk device, for example, once initiated, is processed at the device independently of the CPU. When the device completes the request, it raises an interrupt to a signal the processor that the operation has completed. This signal is treated as a high-priority event: the device is relatively slow compared to the processor, the device needs attention, and some other user may be waiting for the physical device to be free. When the processor recognizes the interrupt request, it:

1. Stops whatever it is doing immediately (unless it is already servicing a higher-priority interrupt request)

2. Saves the status of the current running thread (including the current values of processor registers, e.g. Program Counter showing the next instruction to be executed and the Stack Pointer pointing to the program’s working storage) in an internal data structure called the Thread Context.

3. Begins processing the interrupt.

The thread that was running when the interrupt occurred return to the Ready Queue, and it might not be the thread the Scheduler selects to run following interrupt processing.

Interrupt processing likely adds another thread to the Ready Queue, namely the thread that was waiting for the event to occur. IN Windows, one probable consequence of an interrupt is a reordering of the Scheduler Ready Queue following interrupt processing. The device driver that completes the interrupt processing supplies a boost to the priority of the application thread that transitions from waiting to ready when the interrupt processing completes. Interrupt processing juggles priorities so that the thread made ready to run following interrupt processing is likely to be the highest-priority thread waiting to run in the Ready Queue. Thus, the application thread waiting for an I/O request to complete is likely to receive service at the processor next.

Voluntary wait

A thread voluntarily relinquishes the processor when it issues an I/O request and then waits for the request to complete. Other voluntary waits include a timer wait or waiting for a serialization signal from another thread. A thread issuing a voluntary wait enters the Wait State, causing the Windows Scheduler to select the highest-priority task in the Ready Queue to execute next. The Thread Wait Reason for a thread in a voluntary wait is 7.

Involuntary wait

Involuntary waits are most frequently associated with virtual memory management. For example, a thread enters an involuntary wait if the processor attempts to execute an instruction referencing data in buffer that is currently not resident in memory. Since the instruction cannot be executed, the processor generates a page fault interrupt, which Windows must resolve by allocating free page in memory and reading the page containing the instruction or data into memory from disk. The currently running process is suspended and the Program Counter reset to re-execute the failed instruction. The suspended task is placed in an involuntary wait state until the page requested is brought into memory and the instruction that originally failed is executed. At that point, the VM manager component of Windows is responsible for transitioning the thread from wait state back to ready.

Time allotment exceeded

A thread does not need to perform I/O or wait for an event is not allowed to monopolize the processor completely. Without intervention from the Scheduler, some very CPU-intensive execution threads will attempt to do this. A program bug may also cause the thread to go into an infinite loop, in which it attempts to execute continuously. Either way, the Windows Scheduler eventually interrupts the running thread if no other type of interrupt occurs. If the thread does not relinquish the processor voluntarily, the Scheduler eventually forces it to return to the Ready Queue. This form of processor sharing is called time-slicing, and it is designed to prevent a CPU-bound task from dominating the use of the processor for an extended period of time. Without time-slicing, a high-priority CPU-intensive thread could delay other threads waiting in the Ready Queue indefinitely. The Windows Scheduler implements time-slicing by setting a clock timer interrupt to occur at regular intervals to check on the threads that are running.

When a thread’s allotted time-slice is exhausted, Windows Scheduler interrupts it and looks for another ready thread to dispatch. Of course, if the interrupted thread happens to be the highest-priority thread (or only ready thread), the Scheduler selects it to run again immediately. However, Windows also lowers the likelihood that a CPU-intensive thread will monopolize the processor. This technique of boosting the relative priority of threads waiting on device interrupts and reducing the priority of CPU-intensive threads approximates a mean time to wait algorithm, a technique for maximizing throughput in a multiprogramming environment.

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

General: Understanding Processor - Ready Queue

2007-05-13T22:30:00.001+08:00

Processor Ready Queue

The Processor Queue Length counter in the System object is an extremely important indicator of processor performance. It is an instantaneous peek at the number of Ready threads that are currently waiting to run. Even though reporting processor utilization is much more popular, the Processor Queue Length is actually a more important indicator of a processor bottleneck. It shows that work is being delayed, and the delay is directly proportional to the length of the queue.

Since there is one Scheduler Dispatch Queue that services all processors, the Queue Length counter is only measured at System level. The Thread State counter in the Thread object indicates precisely which threads are waiting for service at the processor(s). In other words, the Processor Queue Length counter indicates how many threads are waiting in the Scheduler dispatch Ready Queue, while the Thread State counter tells which thread are in the queue. A good working assumption is that when the processor is very busy queuing delays impact the performance of executing threads. The longer the queue, the longer the delays that threads encounter.

By monitoring the Thread State counter for all instances of the Thread object at the same time, it is possible to determine which threads are waiting in the Processor Ready Queue at the time measurements are taken. Note that the instantaneous measure of the size of the Ready Queue is influenced by the dispatchability of the performance monitor application itself. In other words, the Ready Queue at the time the Scheduler was finally able to dispatch the System Monitor Measurement thread. The System Monitor runs at high priority, but by no means at the highest priority in the system.

Priority Scheduling is the general solution designed to cope with situations where processor is very busy. Priority scheduling orders the Ready Queue, ensuring that under conditions of scarcity, the highest-priority work gains favored access to the resource.

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

General: Understanding Processor - Processor State

2007-05-13T22:29:00.000+08:00

Processor utilization can be further broken down into time spent executing in user mode (Intel Ring 3) or in privileged mode (Ring 0), two mutually exclusive states. Applications typically run in the more restricted user mode, while operating system functions run in privileged mode. Whenever, an application implicitly or explicitly calls an OS service (e.g. to allocate or free memory, or perform some operation on a file), a context switch occurs as the system transitions from user to privileged mode and back again. The portion of time that a tread is executing in user mode is captured as % User Time; privileged mode execution time is captured in the % Privileged Time counter.

Processor time usage in Windows is broken out into two additional subcategories. % Interrupt Time represents processor cycles consumed in device driver interrupt service routines (ISRs), which process interrupts from attached peripherals such as the keyboard, mouse, disks, network interface card, etc. This is worked performed at very high priority, typically while other interrupts are disabled. It is captured and reported separately not only because of its high priority, but also because it is not easily associated with any particular user process. Windows also tracks the amount of time device drivers spend in deferred procedure calls (DPCs), which also service peripheral devices but run with interrupts enabled. DPCs represent higher-priority work than other system calls and kernel thread activity. Note that % DPC Time is already included in the % Privileged Time measure.

The Scheduler’s thread timing function is notified whenever any context switch occurs, and dutifully records the processing time for the copmpleted function in the appropriate bucket. The context switch might involve going from one user thread to another, a user thread to a kernel function, or a kernel thread to an ISR, followed by a DPC

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

General: Understanding Processor - Thread State

2007-05-13T22:28:00.000+08:00

As we know, a multiprogramming OS switches the processor back and forth between all the program threads that are executing. When the current thread blocks, usually due to I/O, the Windows Scheduler finds another thread that is ready to run and schedules it for execution. If no threads are ready to run, Windows schedules a thread associated with the System Idle process to run instead. When an I/O operation completes, a blocked thread becomes eligible to run again. This scheme means that threads alternate back and forth between the two states: a ready state, where a thread is eligible to execute instructions, and a blocked state. A blocked thread is waiting for some system event that signals that the transition from waiting to ready can occur.

Thread state Counter is an instantaneous counter that you will need to observe at very fine granularity to catch this behavior. The following tables described the thread states and reasons.

Values for Thread State Counter

Value: State
0: Initializing
1: Ready
2: Running
3: Standby
4: Terminated
5: Waiting
6: Transition
7: Unknown

Values for the Thread Wait Reason counter

Value: Reason
0 : Waiting for a component of the Windows NT Executive
1 : Waiting for a page to be freed
2 : Waiting for a page to be mapped or copied
3 : Waiting for space to be allocated in the page or nonpaged pool
4 : Waiting for an Execution Delay to be resolved
5 : Suspended
6 : Waiting for a user request
7 : Waiting for a component of the Windows NT Executive
8 : Waiting for a page to be freed
9 : Waiting for a page to be mapped or copied
10 : Waiting for space to be allocated in the page or nonpaged pool
11 : Waiting for Execution Delay to be resolved
12 : Suspended
13 : Waiting for a user request
14 : Waiting for an event pair high
15 : Waiting for an event pair low
16 : Waiting for an LPC Receive notice
17 : Waiting for an LPC Reply notice
18 : Waiting for virtual memory to be allocated
19 : Waiting for a page to be written to disk
20+ : Reserved

Related Topics

Message: Change of WebSite Template
Message: Some enhancements!
Message: Thank you & Content Page
Welcome!!

Protocols: Working with Web (HTTP/HTML) III - NTLM Authentication

2007-05-13T16:10:00.001+08:00

LoadRunner supports recording and replaying of NTLM authentication. This is described in two main KB articles below. You can refer to the KBs for information on working with it. Take note that you require an account to login to the Mercury/HP support website to view the articles.

Tips on NTLM authentication when working with Web Vusers

How to record against web applications that go through NTLM authentication

In additional to the above, take note that be it any authentication (Basic or NTLM), if it is running over HTTP, it will be captured as it is transmitting in pure text form (except of course the encrypted password).

Therefore, whenever there is problem in the replay, look at the Replay Log. It will show you the transmission sequence of the script. For NTLM, you will be able to see the 4-way authentication process in the Authentication header of the HTTP message. Trace through the sequence of the authentication in the header. If you are unfamiliar with NTLM authentication, it would be good to research here. Also, you can do additional reading on this book, "HTTP Definitive Guide".

In summary, you can start off with the two KBs on information for working with NTLM. Furthermore, a good knowledge of how NTLM works allows you to understand the information in the Replay Log when the script fails.

Products: LoadRunner

2007-05-13T15:09:00.000+08:00

LoadRunner is a load testing tool that allows load testing against various technologies (or protocols to be specified). The load testing is by done generating load of emulated real users in the form of scripts or in the LoadRunner terminology, Vusers. While load generating, monitoring is applied to the SUT (System Under Test) or servers and collected at the end of the load testing. This data, is then been analyzed to determine the next course of action (depending on the objective of the load test, i.e. benchmarking, SLA, stress testing, etc.) such as tuning, scaling, or defining a new SLA, etc.

If you like to know more on the sales talk (again), feel free to explore the HP Software website here.

LoadRunner have a couple of components that makes up the whole product, namely, the Vugen, Controller, Load Generator and Analysis. Take note that Diagnostics and SiteScope is a separate product by itself although it comes along with every LoadRunner CD. It would deserve more description than it will have here. The idea of LoadRunner is fairly simple, it is to generate load based on a described business process, emulated users in the form of ANSI C scripts.

The components described, facilitates the load testing in the following manner. Vugen generates scripts that emulate the real users through recording a described business process that real users would perform. The recording is somewhere like a network sniffer on the protocol described and will generate a basic script that describes the user actions. From here, modifications can be made to the scripts such as runtime settings, parameterization and correlation, and not limited to the two mentioned.

Once the scripts (Vusers) are ready, a scenario is created in the Controller. The scenario defines the number of Vusers (scripts) to generate, the duration of the scenario, the behavior of the Vusers, and the resources (servers) that the tester would like to monitor. It also defines the Load Generators to use and any additional components such as IP Spoofing or Diagnostics module. When the scenario is done, all the tester needs to do is to run the scenario and wait for the results to be collected at the end of the test. More information on how the monitors in LoadRunner work can be found here.

When the scenario is started, the scripts are sent by the Controller to the Load Generator and the Load Generator wills generate the defined assigned amount of load to the SUT. As the name implies, Load Generator generates load and Controller controls the scenario execution.

After the load testing data have been collected. It will be save in a results folder and been port over to Analysis for performing analysis of the results. Analysis is a powerful graphical tool to look at correlations between load and system resource performance (and many more depending on what you are entitled to monitor). However, take note that having a powerful tool does not give the tester wings to fly. The tester will still need the relevant experiences and knowledge to determine bottlenecks in the system. (Usually companies charge this as consultancy).

The above is described in more detail in a popular site describing LoadRunner architecture. If you are interested in the details, please feel free to explore here.

LoadRunner is tied to one license which entitles to the (1) number of Vusers (concurrent users) that are being generated, the (2) protocol that the Controller is able to run and the (3) available monitors that the organization can utilized. This information can be found in the LoadRunner License Information. The restriction of the license only applies to the Controller. Therefore, Vugen and Analysis is still functional even if the license is not applied or expired.

Some pitfalls; There are always questions on client activities been recorded in LoadRunner (Vugen). I believed it’s the association of Functional Testing with the idea of Performance/Load Testing. Take note that they are two different things. Performance/Load Testing involves a lot of knowledge on architecture, applications, etc. It is not as simple in testing GUI on the front end and ensuring that the output is correct.

LoadRunner is designed to load test a designated server and not client. Whatever activities performed by the client is not recorded. This had been a popular question posted frequently on the web and I feel that association should be corrected. For this, I’ve also written an article on it. Click here if you like to know more details.

Installation; for LoadRunner to function properly, it’s advisable to install with the default administrator rights. Installation or functionality problems are usually caused by insufficient privileges to the installed machine, clone images or anti-virus software.

Monitors: Unix System Resource

2007-05-13T13:49:00.000+08:00

Before we start off, a general understanding of how monitoring works in LoadRunner can be found in "Monitors: How does the monitoring work?"will be useful.

Generally, LoadRunner is drawing data from rstatd daemon. For this, before you connect to the Unix box, ensure that rstatd is configured properly and running. This information can be found in the Monitor Reference with every installation of LoadRunner.

To configure rstatd, ensure that you have (1) root account or someone who has root account to assist you and (2) someone who is Unix-trained to assist as much of the configuration is done on the Unix box.

Verify rstatd Daemon

To verify if rstatd is configured, run rup command in the following syntax in the console.

>rup host

You can also use lr_host_monitor and see if it returns any relevant statistics. If the command returns meaningful statistics, the rstatd daemon is already configured and activated. If not, or if you receive an error message, the rstatd daemon is not configured.

Configure rstatd Daemon

To configure the rstatd daemon:

1. Run the command: su root

2. Go to /etc/inetd.conf and look for the rstatd row (it begins with the word rstatd). If it is commented out (with a #), remove the comment directive, and save the file.

3 From the command line, run:

kill -1 inet_pid

where inet_pid is the pid of the inetd process. This instructs the inetd to rescan the /etc/inetd.conf file and register all daemons which are uncommented, including the rstatd daemon.

4 Run rup again.

If the command still does not indicate that the rstatd daemon is configured, contact your system administrator.

Firewall

To monitor a UNIX machine through a firewall, you must run a UNIX utility called rpcinfo and identify the rstatd’s port number. By running rpcinfo -p , you will receive a list of all RPC servers registered in the host’s portmapper, along with the port number. This list will not change until rstatd is stopped and rerun.

Some firewalls allow you to open an RPC program number instead of a port. In such cases, open program 100001. If are prompted to include a version number, specify versions 3 and 4.

Troubleshooting

There are two KBs that discusses about troubleshooting Unix System Resource and would be a good source or starting point for you in the event you run into problems with setting up the Unix System Resource monitoring.

Error: "Monitor name: UNIX Resources. Cannot initialize the monitoring on ..."
Requirements for Unix/Linux Resource Monitor in LoadRunner

You will have to have a valid account with the support site (support.mercury.com) to access the documents.

Products: Performance Center

2007-05-11T13:33:00.000+08:00

Performance Center is an enterprise version of LoadRunner (LR) where it allows concurrent runs in an organization by managing a pool of Controllers and Load generators build on a web interface. The design is for suitable for organizations that require running multiple load test projects throughout the organization. If you like to know more on the sales talk from the company, you can click here.

To put it simply, PC can run multiple load test while LR can run a single load test.

Performance Center tends to design in a way that load tests are managed in the concept of projects. And through these projects, you will be able to trace the number of runs pertaining to each individual.

Privileges are also introduced with the hindsight of projects where there is a difference in the roles. Administrator role maintains the PC setup as a whole, while privileges for users such as Consultant, Tester or Manager can be defined depending on the organization requirements.

The administrator has the rights to manage the resources as well as determine the maximum number of concurrent users the project can generate.

Through PC, a pool of Controllers and Load Generators can be managed together via a single point of control. In this way, resources are better utilized (according to Mercury/HP). Furthermore, there can be configurations to limit the usage of Load Generators to the projects assigned.

License entitlement (or enforcement?) is increased with one more layer of control. the amount of concurrent runs and the total number of vusers that the PC installation can generate. As it is an enterprise version of LoadRunner, the LR license engine still applies, such as the number of vusers, the protocols and monitors entitlement. Both PC and LR licenses are needed in order for the full PC to function properly.

Technically, PC is an enhanced web version of LoadRunner on both JSP and ASP at this point of time. The internal infrastructure is built on DCOM links that needs to exist in the same LAN. Performance Center (PC) maintains a database that keeps information of the load tests as well as the configurations of the PC setup.

The basic infrastructure of Controller and Load Generator still applies even after porting over to this software. Therefore, the troubleshooting, tips and tricks still applies for them.

For the Controllers and Load Generators, they can be located in a separate LAN. PC comes with a nice feature of managing the resources such as patch updates and restarting function over the interface.

Vugen and Analysis remains the same as a client-side program. PC provides these two components by making them downloadable in the PC web pages through a download link. This page, with the right ASP knowledge, can be configured to include other downloads. This, of course is better than LR in terms of installation procedures as you need not go around physically to with the disc to install them.

With all the above mentioned, being an enterprise means more user usage and more features which coherently means more money to pay for by the organization. Therefore, conduct some evaluation before jumping into the decision of purchasing this behemoth product.

To sum up, PC is an enhanced web version of LR with the capabilities of running multiple load tests instead of a single load test. It is managed in projects and can be traced to the number of runs. The licenses have increase to one additional layer in controlling the total number of concurrent runs allowed. Technically, it is build on DCOM and the basic infrastructure of Controller and Load Generator remains.

Scripts: Set Debug Mode in Script

2007-05-11T11:22:00.000+08:00

Sometimes there is a need to know what is the data being transmitted in the Replay Log or performing troubleshooting of the scripts. When the usual options are configured. Parameter substitution, Advance Trace and Data Returned by Server are turned on, it’s going to generate a big chunk of data.

And in circumstances, you need to know only a certain portion of script to ensure that it is working perfectly, you have to wait through the replay, waiting for it to complete. As such, below is a faster way of generating the Replay Log in a shorter format. This, of course is not effective.

So, how do we reduce the information generated to the Replay Log?

You can use the lr_set_debug_message to display only a certain portion of the log generated. Place the lr_set_debug_message before the codes that you want the replay log to receive and close it again as per your requirement. The syntax is as below and more information can be consulted in the Function Reference.

lr_set_debug_message(LR_MSG_CLASS_EXTENDED_LOG LR_MSG_CLASS_RESULT_DATA
LR_MSG_CLASS_PARAMETERS LR_MSG_CLASS_FULL_TRACE, LR_SWITCH_ON);

/*
…. Actions
*/

Lr_set_debug_message(LR_MSG_CLASS_EXTENDED_LOG LR_MSG_CLASS_RESULT_DATA
LR_MSG_CLASS_PARAMETERS LR_MSG_CLASS_FULL_TRACE, LR_SWITCH_OFF);

Scripts: Remove Think Time

2007-05-11T10:42:00.000+08:00

Yesterday, my team got a requirement to randomize the think time and subtract the think time to reflect the actual response time of the transactions of the load test.

To achieve the two requirements, (1) randomize think time and (2) subtract the think time from the transaction response time, two KBs were used.

Randomize the think time with a random parameter described in KB 3448, "How to generate a random think time". Proceed to complete enhancing the script and port it over to the Controller. Run the scenario as per norm
Once the scenario completes and anaylsis graph generated, filter the think time in the Analysis graph to get the processing time. Use KB 13748, "How to exclude think time from transaction timings in the Analysis graphs" to filter out the think time.

Hope this little piece of information is useful.

Message: Some enhancements!

2007-05-11T09:12:00.000+08:00

Hi Visitors,

There are some enhancments to the blog. I've resolved the search box above which will allow searchs on the web and more importantly the blog (a blunder I made on the URL that caused it to fail).

Anyway... new content pages for Message Board for the list of messages and Products for descriptions of products I came across. Lastly, the email subscribing feature if you would like to be updated on any new articles posted here.

If you have any suggestions or comments, feel free to drop me an email at papayamilkshake@gmail.com. Suggestions and comments will be more than welcome! :)

Thank you.

Regards
Hs

Scripts: Auto or Manual Correlation?

2007-05-11T09:04:00.001+08:00

Vugen comes with a nice feature called "Auto-Correlation Studio" in it. This, helps to detect changes on the recorded and replay scripts.

However, sometimes, the auto-correlation does not work as successfully as it is designed. It may not detect over sized dynamic values such as VIEWSTATE (KB 24264) or header information or cookies, etc. Well, to sum up, it varies and may not work.

Manual Correlation is just the good-old method of correlation without any studio gimicks. This although slow, provides you a better "control" of what is needed to be correlated. You can download the document relating to correlation in KB 11806.

My suggestions for those new to correlation:

Do not overly depend on auto-correlation studio
If replay fail, look at the Replay Log to determine the cause. Replay log provides a vast amount of information.
If manual correlation is needed, perform each dynamic value one at a time progressively. In this way, it minimises the risk of missin out any dynamic value.

Content Page - Products

2007-05-11T09:03:00.003+08:00

Performance Center
LoadRunner
Diagnostics
SiteScope
SilkPerformer (a comparison with LoadRunner)

Content Page - Message Board

2007-05-11T09:03:00.001+08:00

Monitors: The Other Way?

2007-05-10T11:41:00.000+08:00

I’ve came across a question on monitoring Solaris boxes for a 99.99% uptime for 7 days in a forum. In LoadRunner context, it will be considered to be Unix System Resource monitoring. There were suggestions to monitor via BAC (Business Availability Center) rather than using LoadRunner. Actually it goes down to the purpose of what the person is trying to achieved.

LoadRunner is designed to monitor during the period of load test while BAC is designed to monitor in a production environment; which relates to two questions:

1. Are you monitoring over a period of load? Or,
2. Are you monitoring in a production environment?

Anyway, I’ve came out with three suggestions in respond to his enquiry.

1. Use a single Vuser to monitor the Unix system resource for the 7 days period.
2. Use BAC to monitor.
3. Use "uptime" command in the Unix box and output to a log file periodically.

Let’s discussed in details each of the methods.

[1] Use a single Vuser to monitor the Unix system resource for the 7 days period.

This is a method to circumvent the conventional monitoring technique. You will get your graphs from Unix however, take note that this is not so all perfect. When LoadRunner completes a scenario execution (load test), it collates the results from the Load Generators (LG in short) back to the Controller at the end of the test. Putting it in the 7 days context, there will be 7 days worth of monitoring data that needs to be transmitted back to the Controller.

For this, it may (1) fail transmitting back due to the size of the data over the bandwidth of the organization network, also (2) even when it is successfully sent back to the Controller, Controller requires significant resources to process the data and may end up crashing or hanging.

[2] Use BAC to monitor

BAC and SiteScope, both from Mercury/HP will be able to achieve that. The main purpose of the above products is to monitor in a production environment and send reports or alerts depending on the configuration. There is also other monitoring products out in the market but I guess it’s up the organization to decide what is best for them.

Take note that the monitoring principle for LoadRunner and BAC/SiteScope is the same (BAC is tapping to capabilities of SiteScope monitoring). I’ve discussed that in an article, "Monitors: How does the monitoring work?" previously, which you may want to explore.

[3] Use "uptime" command in the Unix box and output to a log file periodically (Unix only)

This is by far the cheapest method as you are riding on the capabilities of your current resources. Uptime provides information about the system availability since it’s last boot up. For more information of uptime, click here. Also, refer to this resources for redirection output information.

By running a cron job periodically, example every 30 min, and redirect the output to a log file, you can trace when the system was rebooted. For more information of cron job, click here.
Use the following syntax in the crontab file if you are keen implementing this suggestion.

Example:
30 18 * * * uptime >> uptime.log

Message: Thank you & Content Page

2007-05-09T17:52:00.000+08:00

Hi Visitors,

Thanks for the support and comments. :) I will add in more information that I come across and to share with you guys.

As the site is filled with more information, it is becomeing a little messy. So, I've come out with the Content Page on the right hand corner for you to access the articles with less trouble. Do revert if there are missing links or any improvement areas.

Thank you. Happy reading!

Regards
Hs

Content Page - General

2007-05-09T17:38:00.000+08:00

Content Page - Tools

2007-05-09T17:33:00.001+08:00

Tools: Duplicating Files Prior Load Test
Tools: Determine the Windows patches installed using Windows 2003 Resource Kit
Tools: Monitoring Unix Resource Script ver 1.0

Content Page - Scripts

2007-05-09T17:25:00.001+08:00

Scripts: Duplicating Files on the Fly
Scripts: Step Download Timeout
Scripts: VIEWSTATE
Scripts: Auto or Manual Correlation?
Scripts: Remove Think Time
Scripts: Set Debug Mode in Script
Scripts: Replay Failure – Use Full Extended Log!
Scripts: Starting a new transaction during iterations
Scripts: Any compatibility issues after upgrading LoadRunner versions?
Scripts: Parameterization

Content Page - Monitors

2007-05-09T17:20:00.001+08:00

Monitors: How does the monitoring work?
Monitors: What metrics/counters to monitor for Windows System Resource?
Monitors: The Other Way?
Monitors: The Other Way? II
Monitors: Unix System Resource
Monitors: MS SQL Server
Monitors: Memory Glossary

Content Page - Protocols

2007-05-09T16:30:00.000+08:00

General
What Protocol to Use?

Web (HTTP/HTML)
Working with Web (HTTP/HTML)
Working with Web (HTTP/HTML) II - Differences in WinINet and Socket
Working with Web (HTTP/HTML) III - NTLM Authentication
Working with Web (HTTP/HTML) IV - Internet Explorer 7 Support
Working with Web (HTTP/HTML) V - File Download/Export

Java
Working with CORBA-Java
Working with RMI-Java
Working with Java - Introduction
Protocols: Java Error - java.lang.NoClassDefFoundError

Citrix
Working with Citrix
Working with Citrix II - Best Practices by LoadTester

Winsock
Working with Winsock by LoadTester

Protocols: Working with Web (HTTP/HTML) II - Difference between WinINet and Socket Level

2007-05-09T16:07:00.000+08:00

The Windows Internet (WinInet) application programming interface (API) enables applications to interact with Gopher, FTP, and HTTP protocols to access Internet resources. As standards evolve, these functions handle the changes in underlying protocols, enabling them to maintain consistent behavior. More information can be found at Microsoft.

Whereas for Socket Level, it is a proprietary API that Mercury/HP LoadRunner implements. So what does it mean by setting the recording level in Port Mapping of the Recording Options and replay for RTS (Runtime Settings)?

By configuring Vugen record at WinINet level, simply means to instruct Vugen to record using the WinINet API. Vice versa, by configuring the Socket level, you are instructing Vugen to record using the proprietary API. By default, it is Socket level.

For replay, the principle applies were WinINet will be replaying using WinINet API else it will be replaying via the default Socket level API. There is a small description of "What is "WinINet instead of Sockets" for" at the support website that reiterates the above but worth taking a look. A valid account will be required to login to access the article.

When do you configure WinINet or Socket? Sometimes web applications may be designed in WinINet while sometimes it does not require WinINet as often and Socket level will be sufficient. This is a little tricky here but it will most probably end up trial-and-error. When an application fail in recording, you can try by switching between the two modes. Whereas when the application fails in replay, try switching the modes also.

For replay failure, you can try the suggested solution in Protocols: Working with Web (HTTP/HTML) where there is a section on the switching of the replay mode.

Take note that IE browser API may change over time due to the upgrade of versions (e.g. IE 6 to IE 7) which may inherently cause the replay to be successful or fail.

General: Detecting Memory Leaks Using LoadRunner

2007-05-08T10:50:00.001+08:00

How to effectively use LoadRunner to detect memory leaks? First, I recommend you to read the article “Detecting Memory Leaks” that I’ve extracted from the “Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos” to understand the definition of memory leak.

Here, as we discussed memory leak, we are referring the memory leakage on the server side NOT the client side. Therefore, if you are searching for information on client-side application memory leakage, you can try the SQA Forums which may provide more relevant information to you.

With the mentioned materials, we can use the Available Bytes, Committed Bytes and the number of Running Vusers to determine if memory leakage occurred.

Setup the scenario with Available Bytes and Committed Bytes as one of the counters to monitor. Run the load test as per norm. Observe the pattern of memory usage on the designated server with the ramp up of Vusers and see if they correlate.

When the memory usage peaks at a ceiling, exit the Vusers gradually. If there wasn’t any leakage, the Available Bytes should increase and Committed Bytes decreased in correlation with the Vusers exiting. If it doesn’t, that is, the memory usage is persisted to be high, and the Vusers have successfully exited. It may indicate the memory leak existed in the application and will needed to be resolved (usually at the code-level). Code profiling tools such as Parasoft Jtest (etc.) will be useful in detecting potential poor coding standards.

For ASP applications using IIS, memory leak could be indicated when the IIS worker process restarts itself at a defined memory level. However, point to note that as this configuration can be “human-subjective”, the IIS may merely performing what it is supposed to do despite the memory usage pattern.

Recommendation:

I would encourage you to use the search box on the top-right hand corner should you need to search for additional information.

Related Topics: