Performance Testing, LoadRunner Tips&Tricks

LoadRunner TnT

2007-10-29T21:57:00.000+08:00

Just to bring to your attention that if you had not subscribed to the contents of my new site, now is the time to do so. I'm still using FeedBurner to deliver quality LoadRunner and Performance Testing content to you. I'm also including information to analyze, troubleshoot and tune whenever I have anything to share with you guys out there. So why wait any further if you are new to this field, take the opportunity to subscribe to the new site now! :) It's free anyway. :)

It's a brand new start for LoadRunner TnT

2007-09-06T19:58:00.000+08:00

Many thanks to all who had been following and supporting this website. The time to move to a bigger space has come and we are excited to announce the new website LoadRunner TnT is up and running.

Log on to LoadRunner TnT now!

The new site will continue to provide quality information on LoadRunner and Performance Testing to help newbies or mid-career performance tester in their work. Posting on this site will be discontinued but for those existing articles, they will be moved to the new home not shortly but surely.

General: Load Testing Objectives

2007-08-27T09:40:00.000+08:00

There are two approaches from the user perspective on defining load testing objectives, either top-down (business perspective) or bottom-up (techincal perspective).

The top-down, meaning the transaction time take to submit a request and having it responded on the user's machine. This is a business perspective where they just want to know how the application is responding. The user do not want to know how many Hits/sec or Throughput but just want to know how much time was taken for a transaction (therefore the terminology of "transcation" must be aligned with you and the user).

The bottom-up, is a techincal perspective as you had pointed out such as the ASP.NET transactions or the database transactions (or commits). This is usually a persepctive from a technical manager so they can configure the system accordingly (or whatever they are going to do their system).

Hopefully the above is helpful in understandin your users/clients a little bit more before giving them a satisfying answer.

Scripts: Parameterization

2007-08-21T20:14:00.000+08:00

Point to note that parameterization may result in data dependency which the tester should be aware of and handle it appropriately. Every load test should at least require parameterization. Parameterization includes passing different types of data into the application to emulate the real world users performing/entering different values.

What do clients want to achieve from parameterization?

To test different data
Just to load/emulate the data

Parameterization requires identifying with the clients which areas required data input. This will be useful when discussing in the Application Design. From there, they can prepare the data from the database which is a more efficient method.

Preparing the data

Usually after walking through the application, I will request the client to prepare the date via the database and export it out as an excel file. This will reduce the time in preparing lengthy excel or notepad files. From there, you can manipulate the data easily in either Excel or OpenOffice.org. After I’ve amended the files properly, I will save it as a .dat file and placed it in a shared folder for my scripts to access.

Once converted into .dat file, ensure that they have been delimited properly with the commas. Take note that Vugen allows different delimiters in the Parameter List.

The usual items to parameterize are username, password, dates, etc but not limited to the mentioned. However, there are also hidden values that are stored in web pages (e.g. in the form of AJAX) or HTML codes (e.g. hard-coded hidden input) which are captured by Vugen. As such, I will point it out to the clients and request them to prepare the data for me.

Having said that, it’s best to define what is correlation and parameterization or you may end up spending effort in the wrong track.

Placement of the Parameter file

When you created a parameter for each script, by default, it’s stored in the root of the script folder. That is, if you save the script as test_script.usr in C:\, the parameter file will be saved in C:\test_script. For every script that has a parameter, it will be saved in its individual script folder. This way is good for a single script but when multiple scripts using the same parameter file is involved in the load test, it will be advisable to centralize the parameter file.

What I do when I’ve prepared the data files is that I will place them in a central repository (folder) where all my scripts will access. This is tidier and provides a way of ensuring consistency of the parameter used for all scripts.

Why parameterize different data?

Why is there different effect on different data? A simple example is uploading of different sizes of files. If parameter A is uploading 2MB of file while parameter B is uploading 4MB of file, it will definitely generate a higher throughput on the latter parameter. This may be a concern of the client.

Another example is querying different items in a database. Parameter A may query and return a smaller result while parameter B returns a bigger result causing a higher throughput.

Not the Obvious Data Dependency

Point to note that parameterization may result in data dependency which the tester should be aware of and handle it appropriately.

Examples are username and password submission: A username login will require a valid password. This may seemed obvious for this situation. However, as pointed earlier about the hidden input, you will have to be cautious on it. As Vugen records whatever is been transmitted to the server, the hidden input are recorded as well. Therefore, you will need to check back with the application developers to find out more about the hidden values associated with each parameter value. If possible, request them to provide a list of the values in Excel file format for you to manipulate.

Which Settings are correct?

Parameterization settings vary depending on your applications. You may like to sequential it or randomize it. Ensure the settings are correct. Usually I defined them as unique.

Testing the data

There is various ways to test if the parameters are correct. You can perform the following:

Replay the script in Vugen with one iteration
Replay the script in Vugen with three iterations.
Replay the script in Controller with one Vuser.
Replay the script in Controller with three Vusers.

These will be sufficient to test out the validity of the parameters used. You can also defined at which point the parameter values are been used by defining the starting row to retrieve data.

Other tips & tricks

Some applications/business processes only allow one time use of the parameter. For example, deletion. As such, you should be aware that after each test, request the client or the application team to revert the changes for you so that there are data for you to delete.

Related Topics

Scripts: Duplicating Files on the Fly
Scripts: Step Download Timeout
Scripts: VIEWSTATE
Scripts: Auto or Manual Correlation?
Scripts: Remove Think Time
Scripts: Set Debug Mode in Script
Scripts: Replay Failure – Use Full Extended Log!
Scripts: Starting a new transaction during iterations
Scripts: Any compatibility issues after upgrading LoadRunner versions?

Scripts: Any compatibility issues after upgrading LoadRunner versions?

2007-08-19T13:20:00.000+08:00

Personally, I encountered problem while porting the scripts recorded in 8.0 to 8.1 . This was actually pertaining to Citrix protocol where there is an additional parameter needed for the APIs. This will be particular to users that are using scripts in BPM (Business Process Management) of BAC (Business Availability Center). Take note that BPM uses the same recording tool, VuGen in LoadRunner, however at a version lower than the currently released LoadRunner Vugen).

If I didn't recall wrongly, the Citrix API ctrx_sync_on_bitmap, had an additional parameter, "CTRX_LAST" at the end of its parameter list in 8.1 as compared to 8.0. Since then, I haven't had any problem with other protocol scripts so far.

Anyway, my advice are as the followed:

Read the upgrade readme file.
Clone an image prior the upgrade. In this way, you can revert back to the older version.
Test the previously recorded scripts from the older version.

Do you have any incompatibility issues of scripts before? Feel free to share with us!

Understanding Network: Performance Measurements

2007-08-15T13:05:00.000+08:00

Evaluating path performance means doing three types of measurements. measurements will give you an idea of the hardware capabilities of your network, such as the maximum capacity of your network. BandwidthThroughput measurements will help you discover what capacity your network provides in practice, i.e., how much of the maximum is actually available. Traffic measurements will give you an idea of how the capacity is being used.

Performance Measurements

Two factors determine how long it takes to send a packet or frame across a single link. The amount of time it takes to put the signal onto the cable is known as the transmission time or transmission delay. This will depend on the transmission rate (or interface speed) and the size of the frame. The amount of time it takes for the signal to travel across the cable is known as the propagations time or propagations delay. Propagation time is determined by the type of media used and the distance involved.

Once we move to multi-hop paths, a third consideration enters the picture – the delay introduced from processing packets at intermediate devices such as routers and switches. This is usually called the queuing delay since, for the most part, it arises from the time packets spend in queues within the device. The total delay in delivering a packet is the sum of these three delays. Transmission and propagation delays are usually quite predictable and stable. Queuing delays, however, can introduce considerably variability.

The term bandwidth is typically used to describe the capacity of a link.

Throughput is a measure of the amount of data that can be sent over a link in a given amount of time. Throughput estimates, typically obtained through measurements based on the bulk transfer of data, are usually expressed in bits per second or packets or second. Throughput is frequently used as an estimate of the bandwidth of a network, but bandwidth and throughput are really two different things. Throughput measurement may be affected by considerable overhead that is not included in bandwidth measurements. Consequently, throughput is a more realistic estimator of the actual performance you will see.

Throughput is generally an end-to-end measurement. When dealing with multi-hop paths, however, the bandwidths may vary from link to link. The bottleneck bandwidth is the bandwidth if the slowest link on a path, i.e., the link with the lowest bandwidth.

The above was extracted from the book, "Network Troubleshooting Tools" by Joseph D. Sloan.

Related Topics

Content Page - General

Understanding Network: How traceroute works?

2007-08-12T21:51:00.000+08:00

The program was written by Van Jacobson and others. It is based on a clever use of the Time-To-Live (TTL) field in the IP packet’s header. The TTL field is used to limit the life of a packet. When a router fails or is mis-configured, a routing loop or circular path may result. The TTL field prevents packets from remaining on a network indefinitely should such a routing loop occurs. A packet’s TTL field is decremented each time the packet crosses a router on its way through a network. When its value reaches 0, the packet is discarded rather forwarded. When discarded, the ICMP TIME_EXCEEDED message is sent back to the packet’s source to inform the source that the packet was discarded. By manipulating the TTL field original packet, the program traceroute uses information from these ICMP messages to discover paths through a network.

Traceroute sends a series of UDP packets with the destination address of the device you want a path to. * By default, traceroute sends sets of three packets to discover each hop. Traceroute sets the TTL field in the first three packets to a value of 1 so that they are discarded by the first router on the path. When the ICMP TIME_EXCEEDED messages are returned by that router, traceroute records the source IP address of these ICMP messages. This is the IP address of the first hop on the route to the destination.

Next, three packets are sent with their TTL field set to 2. These will be discarded by the second router on the path. The ICMP messages returned by this router reveal the IP address of the second router on the path. The program proceeds in this manner until a set of packets finally has a TTL value large enough so that the packets reach their destination.

Typically, when the probe packets finally have an adequate TTL and reach their destination, they will be discarded and an ICMP PORT_UNREACHABLE message will be returned. This happens when traceroute sends all its probe packets with what should be invalid port numbers, i.e., port numbers that aren’t usually used. To do this, traceroute starts with a very large port number, typically 33434, and increments this value with each subsequent packet. Thus, each of the three packets in a will have three different unlikely port numbers. The receipt of ICMP PORT_UNREACHABLE messages is the signal that the end of the path has reached.

Should a packet be lost, an asterisk is printed in the place of the missing time. In some cases, all three times may be replaced with asterisks. This can happen for several reasons. First, the router at this hop may not return ICMP TIME_EXCEEDED messages. Second, some older routers may incorrectly forward packets even though the TTL is 0. Third possibility is that ICMP messages may be given low priority and may not be returned in a timely fashion. Finally, beyond some point of the path, ICMP packets may be blocked.

Options

-n: disable name resolution.

-v: enable verbose option which will log source and packet sizes of the probes will be reported for each packet.

-m: define maximum number of hops where default is 30 hops before halting.

-p: traceroute usually receives a PORT_UNREACHABLE message when it reaches its final destination because it uses a series of unusually large port numbers as the destination ports. Should the number actually match a port that has a running service, the PORT_UNREACHABLE message will not be returned. This is rarely a problem since three packets are sent with different port numbers, but, if it is, the option lets you specify a different starting port so these ports can be avoided.

-q: traceroute sends three probe packets for each TTL value with a timeout of three seconds for replies. This can be changed using –q option.

-w: define the default timeout value for the probe packets.

The above was extracted from the book, "Network Troubleshooting Tools" by Joseph D. Sloan.

Related Topics

Content Page - General

Understanding Network: How Ping Works?

2007-08-09T11:28:00.000+08:00

One network device sends a request for a reply to another device and records the time the request was sent. The device receiving the request sends a packet back. When the reply is received, the round-trip time for packet propagation can be calculated. The receipt of a reply indicates a working connection. This elapsed time provides an indication of the length of the path. Consistency among repeated queries gives an indication of the quality of the connection. Thus, ping answers the two basic questions. Do I have a connection? How good is that connection?

Clearly, for the program to work, the networking protocol must support this query/response mechanism. The ping program is based on Internet Control Message Protocol (ICMP), part of the TCP/IP protocol. ICMP was designed to pass information about network performance between network devices and exchange error messages. It supports a wide variety of message types, including query/response mechanism.

The normal operation of ping relies on two specific ICMP messages, ECHO_REQUEST and ECHO_REPLY, but it may response to ICMP messages other than ECO_REPLY when appropriate. In theory, all TCP/IP-based network equipment should respond to an ECHO_REQUEST by returning the packet to the source, but this is not always the case.

Interpreting Results

In different flavors of ping, results vary. However, for each packet we are given the size and source of each packet, an ICMP sequence number, a Time-To-Live (TTL) counter, and the round-trip times. Of course, the sequence number and the round trip time are the most revealing when evaluating basic connectivity.

When each ECHO_REQUEST packet is sent, the time the packet is sent is recorded in the packet. This is copied into the corresponding ECHO_REPLY packet by the remote host. When an ECHO_REPLY packet is received, the elapsed time is calculated by comparing the current time to the time recorded in the packet, i.e., the time the packet was sent. This difference, the elapsed time, is reported along with ECHO_REPLY packet is received that matches a particular sequence number, that packet is resumed lost. The size and the variability of elapsed times will depend on the number and speed of intermediate links as well as the congestion on those links.

It may seem that the TTL field could be used to estimate the number of hops on a path. Unfortunately, this is problematic. When a packet is sent, the TTL field is initialized and is subsequently decremented by each router along the path. If it reaches zero, the packet discarded. This imposes a finite lifetime on all packets ensuring that, in the event of a routing loop, the packet won’t remain on the network indefinitely. Unfortunately, the TTL field may or may not be reset at the remote machine and, if reset, there is little consistency in what it is set to. Thus, you need to know very system-specific information to use the TTL field to estimate the number of hops on a path.

Options

-c: allow you to specify the number of packets you want to send.
-f: used to flood packets onto network. This option is to send as fast as the receiving host can handle them which is useful for stress testing a link or to get some indication of the comparative performance of interfaces. This is restricted to root.
-l: used to flood packets onto network. It takes a count and sends out that many packets as fast as possible which eventually falls back to normal mode. This could be used to see how the router handles a flood of packets. This is restricted to root.
-i: allows the user to specify the amount of time in seconds to wait between sending consecutive packets.
-n: restricts output to numeric form which is useful if you have DNS problems.
-v: used for verbose output.
-q, -Q: used for quiet output.
-s: specifies how much data to send. If set too small, less than 8, there won’t be space in the packet for a time-stamp. Setting the packet size can help in diagnosing a problem caused by path Maximum Transmission Unit (MTU) settings (the largest frame size that can be sent on the path) or fragmentation problems. (Fragmentation is dividing data among multiple frames when a single packet is too large to cross a link. It is handled by the IP portion of the protocol stack.) The general approach is to increase packet sizes up to the maximum allowed to see if at some point you have problems. When this option isn’t used, ping defaults to 64 bytes, which may be too small a packet to reveal some problems. Also, remember that ping does not count the IP or ICMP header in the specified length so that your packets will be 28 bytes larger than you specify.

You could conceivably see MTU problems with protocols, such as PPP, that use escaped characters as well. With escaped characters, a single character may be replaced by two characters. The expansion of escaped characters increases the size of the data frame and can cause problems with MTU restrictions or fragmentation.

-p: allows you to specify a pattern for the data included within the packet after the timestamp.

The above are not the entire list of options. As such, be sure to consult the documentation if things don’t work as expected.

Using Ping

To isolate problems with ping, you will want to run it repeatedly, changing your destination address so that you work your way through each intermediate device to your destination. You should begin with your loopback interface. Use either localhost or 127.0.0.1. Next, ping your interface by IP number. (Run ifconfig –a if in doubt.) If either of these fails, you know that you have a problem with the host.

Next, try a host on a local network that you know is operational. Use its IP address rather than its hostname. If this fails, there are several possibilities. If other hosts are able to communicate on the local network, then you likely have problems with your connection to the network. This could be your interface, the cable to your machine, or your connection to a hub or switch. Of course, you can’t rule out configuration errors such as media type on the adapter or a bad IP address or mask.

Next, try to reach the same host by name rather than number. If this fails, you almost certain to have problems with name resolution.

Try reaching the near and far interfaces of the router. This will turn up any basic routing problems you may have on your host or connectivity problems getting to your router.

If all goes well here, you are ready to ping remote computers. (You will need to know the IP address of the intermediate devices to do this test. If in doubt, use traceroute to determine the machines.) Realize, of course, that if you start having failures at this point, the problem will likely lie beyond your router. For example, your ICMP ECHO_REQUEST packets may reach the remote machine, but it may not have a route to your machine to use for the ICMP ECHO_REPLY packets.

When faced with failure at this point, your response will depend on who is responsible for the machines beyond your router. If this is still part of your network, you will want to shift your tests to machines on the other side of the router and try to work in both directions.

If these machines are outside your responsibility or control, you will need to enlist the help of the appropriate person. Before you contact this person, you should collect as much as information as you can. There are three things you may want to do. First, go back to using IP numbers if you have been using names. As said before, if things start working, you have a name resolution problem.

Second, if you were trying to ping a device several hops beyond your router, go back to closer machines and try to zero in on exactly where you first encountered the problem.

Finally, be sure to probe form more than one machine. While you may have a great deal of confidence in your local machine at this point, your discussion with the remote administrator may go much more smoothly if you can definitely say that you are seeing this problem from multiple machines instead of just one.

Problems with Ping

The program does not exist in isolation, but depends on the proper functioning of other elements of the network. Ping usually depends upon ARP and DNS. As previously mentioned, if you are using a hostname rather than an IP address as destination, the name of the host will have to be resolved before ping can send any packets. You can bypass DNS by using IP address.

It is also necessary to discover the host’s link level address for each host along the path to the destination. Although this is rarely a problem, should ARP resolution fail, then ping will fail. You could avoid this problem, in part; by using start ARP entries to ensure that the ARP table is correct. A more common problem is that the time reported by ping for the first packet sent will often be distorted since it reflects both transit times and ARP resolution times. On some networks, the first packet will often be lost. You can avoid this problem by sending more than one packet and ignoring the results for the first packet.

The above was extracted from the book, "Network Troubleshooting Tools" by Joseph D. Sloan.

Related Topics

Content Page - General

General: Planning for Load Testing - Recommendations

2007-08-07T09:58:00.000+08:00

After identifying trends, it’s time to propose an action plan, proposal or recommendation to your client.

This maybe just telling the client that “Dear client, your application is running at this rate, I would recommend to re-look the SLAs with your users.” or “Dear client, the server housing the application is encountering processor contention. I would like to recommend you to distribute the load across with a new server (or processor).” This is of course tied to the initial requirements and the limitations the clients are bounded in.

At this point if they have follow up actions to improve the system, do keep a record of the last changed parameter on the server. This will be useful for you to track the changes over time on their application and determined if the changed had enhanced the application. Unless you are part of the server team, you might not be involved in the tuning process.

Once the changes have been made, conduct a load test to verify the changes.

General: Planning for Load Testing - Analyzing

2007-08-05T12:37:00.000+08:00

This is the stage where it intertwined with the monitoring. At this stage, similarly to Monitoring, it is important to have a sound knowledge of the server or database internal workings. With that you will know what to further investigate and what the graph represents.

A graph looks great with the lines flying up and down but that is information. You will need to have a lot of experience to translate that information to useful knowledge. This will be done with merging and correlating of the load generated against the utilization of resources. You must be able to “see” the trend and identify possible bottlenecks.

I would recommend acquiring lots of information of the server (either the software or hardware perspective) you are monitoring as this is required for your analyzing. You can refer a previous consolidated article, "Content Page - General" which provides a proportion of basic information that will be useful to you in facilitating the analyzing.

Other than the knowledge, you will have to refer back to the initial requirements set down by your clients. They can comprised from a top-down interest/approach such as transaction response time to a bottom-up technical interest/approach such as the utilization of the server/network resources. In any case, show them what you've gathered in their best interest.

From there, you will know what to look out for in the graphs, such as amount of users that were actually generated and if the application was able to support the defined transaction response time. Or the utilization across all servers met (either under-utilized, over-utilized or unevenly balanced).

Some combination of graphs that will be useful in analyzing are as followed (but not limited to the list):

Vuser - System Resource (Windows or Unix) or any other resource graphs

Useful in determining/describing the capacity that the resource (server) can work with the generated load.

Vuser - Transaction Response Time

Useful in determining/describing the transaction response time over time when vusers are generated into the application. This will determine the point or amount of users that the server can handle with respect to the defined transaction response time.

Vusers - Transactions Per Seconds

Useful in determining/describing the amount of transactions that were generated with respect to the amount of users generated.

Vusers - Errors

Useful in determining/describing the breaking point of the application where transactions start to fail (consistently) with respect to the amount of users generated.

Vusers - Errors - System Resource

Useful in determining/describing if the resources (servers) are experiencing difficulties handling the generated load with respect to the amount of errors.

Transaction Response Time (Percentile)

Useful in determining/describing the overall performance of the transactions in terms of percentage.

System Resource - System Resource

Useful in determining if the resources (servers) have been properly distributed in load. Usually, I will also ask my clients to verify the load balancer logs and the application logs on the activities.

Please feel free to add or comment if you have a good combination of graphs!

General: Planning for Load Testing - Monitoring

2007-08-03T10:01:00.000+08:00

I separated the Monitoring from the Monitor Setup because I feel that the setup is prior the load test runs. In this stage, what we are concerned are the counters and metrics to monitor. If you are starting off without any knowledge or history of the system such as performance issues, it’s advisable to monitor the 4 main categories, namely Processor, Memory, Disk and Network. Once the load test completes, you should have an overview of the application performance. From there, go to a 2nd level of monitoring till the nth level until you are satisfied with the investigation.

For example, on the 1st load test, you observed that the page faults are occurring at a level higher than the accepted threshold indicating memory problems. From there, you may want to investigate further on the cause of the page fault such as, if it is a hard page fault or soft page fault. Therefore, you may like to monitor the Transition Faults, Pages Input and Pages Output to answer your doubts.

Of course, to know what to investigate further requires knowledge of the application, server, network, and what the counters and metrics mean. This is also assisted if you have a good understanding of the internals working of the component.

Related Topics

General: Planning for Load Testing

General - Content Page

Content Page - Monitors

Monitors: Memory Glossary

2007-08-01T13:33:00.000+08:00

This is a compilation of memory-related counters. Do feel free to revert or comment should you have additional information on the counter.

Available Bytes

Process Working Set growth becomes constrained when Available Bytes <>

Pool Nonpaged Bytes

The system’s non-pageable (fixed) memory.

Pool Paged Resident Bytes

The OS‘s pageable memory that is currently resident in RAM.

System Code Resident Bytes

[Coming soon]

System Driver Resident Bytes

Total System Resident Bytes = Pool Nonpaged Bytes + Pool Paged Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes + System Cache Resident Bytes.

System Cache Resident Bytes

The current amount of RAM used for the file cache.

Page Faults/sec

Can be a grossly misleading number. Page Faults/sec = “soft” Transition Faults + application file Cache Faults + demand zero faults + hard page faults.

Page Reads/sec

This counter measures the number of requests to the I/O manager to retrieve pages of memory from the disk. Despite the name of this counter, it measures requests, not pages; a request can be for more than a single page.
Equivalent to hard page fault rate.

Page Writes/sec

This counter measures the number of requests to the I/O manager to write pages of memory to the disk. Again, each request can be for more than a single page.
Updated “dirty” pages must be flushed to disk before they can be reused by a different application.

Pages Input/sec

This counter measures the number of pages read from the disk per second. Combing this counter with the Page Reads/sec counter can tell you how many pages are retrieved per request.
Calculate the bulk paging rate: Pages Input/sec / Pages Read/sec

Pages Output/sec

This counter measures the number of pages written to the disk per second. Combining this counter with the Page Writes/sec counter can tell you how many pages are written per request.
Try to limit Pages Input/sec + Pages Output/sec to 10 – 20% of total disk bandwidth, if possible. Disk bandwidth absorbed for paging operations is unavailable for application processes.

Pages/sec

This counter measures the total number of pages read and written to the disk. This counter represents the sum of Pages Input/sec and Pages Output/sec. Using this counter along with the Disk Bytes/sec counter from the Physical Disk Object, you can determine what portion of the data transferred to the disk is due to memory access, and what portion is due to file system access.

Committed Bytes

Represents virtual memory pages backed in either RAM or secondary storage (paging files). Calculate a virtual: real memory contention index = Committed Bytes / Total RAM. Consider adding RAM when this ration starts to approach 2:1.

Commit Limit

Maximum number of virtual memory pages that can be allocated without extending the paging file(s).

% Committed Bytes in Used

Committed Bytes / Commit Limit. Consider adding RAM when consistently > 70% on a server.

Cache Bytes

Actually, the system address space working set, but includes the file cache. The sum of Pool Paged Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes + System Cache Resident Bytes.

Transition Faults/sec

“Soft” page faults resolved without having to access the disk.

Cache Faults/sec

Normal application file I/O operations are diverted to use the paging subsystem. Each file cache fault leads to a physical disk read I/O operation.

Demand Zero Faults/sec

The rate at which applications require brand new pages.

Write Copies/sec

Private Copy on Write pages from shared DLLs.

Pool Paged Bytes

Calculate a virtual: real memory contention index = Pool Paged Resident Bytes / Pool Paged Bytes. Compared to Page Reads/sec to anticipate real memory bottlenecks.

General: Planning for Load Testing - Monitor Setup

2007-07-30T22:25:00.000+08:00

The setting up of monitors poses some challenge and I advised you to cater one or two days to implement the monitors. LoadRunner features agentless monitoring (which other products do too). However, in order to achieve such feature it needs the SUT (System Under Test) to be configured properly.

This “configure properly” becomes sort of “out-sourced” to the system admin team and if problem arises, the blame of product issue will be reduced. Depending on the type of monitors you are implementing, you will require time and effort to clear ports and justify your case to the security or the network team before the actual implementation is possible.

This is where the initial gathering of information is important in Soliciting Requirements. With the architecture diagram, you can relate to the security or network team on the team of installation of agents or opening of ports. But of course this will come under scrutiny if you do not provide sufficient convincing information to them.

This is also the part of load test where you start to work with even more people from other teams, such as, system, network, database, security or whatever titles they are given.

Also, take note that for different applications or servers, configuration has to be done before the tool can collect monitoring information from them. Examples like perfmon for Windows, rstatd for UNIX, JMX or JMS for WebLogic.

Point to take note at this stage, every organization has different requirements in their infrastructure and for this, you need to respect that and be sensitive to it.

Related Topics

General: Planning for Load Testing

General - Content Page

Content Page - Monitors

General: Planning for Load Testing - Protocols

2007-07-25T10:04:00.000+08:00

Load testing is about capturing the traffic sent by the application to the server. And using the captured traffic, the tool scripts out APIs to generate the traffic, which is been emulated during replay and sent back to the server.

Through understanding the Application Design, you should be able to determine the type of protocol that is been communicated. As I pointed in previous articles , LoadRunner focuses a lot on protocol and therefore the need to know this information is critical.

To be able to read the logs and find meaningful information out of it, I would recommend you to acquire as much knowledge as possible. For example, HTTP, the replay log merely displays the transmission of the HTTP requests in text. From there, you will know information such as authentication headers or the body that was sent.

I cannot emphasize again that do not associate functional testing as load testing. Therefore, if you are from the functional testing background, please be sure to acquire more knowledge on the protocol that you are working on before proceeding.

Related Topics

General: Planning for Load Testing

General - Content Page

General - Protocols

Understanding Memory: Virtual Addressing (Illustration)

2007-07-23T09:26:00.001+08:00

The following illustration explains the process of virtual addressing when a program is loaded into memory. For detailed explanation, refer to "Understanding Memory: Virtual Addressing".

Executable program image file loads into memory.
Logical memory address range of the application is divided into fixed-size chunks caled pages.
Page mapping in Virtual Memory is dynamic.
Frequently referenced logical addresses tend to reside in physical memory.
Infrequently referenced pages are relegated to paging files on secondary disk storage.

General: Planning for Load Testing - Application Design

2007-07-22T17:37:00.000+08:00

This is the point that the load tester must understand how the application works. When we refer to how it works, it’s not about just clicks on links and buttons, or the calendar will pop up. It goes deeper such as the following but not limited to the list:

What is the type of communication between the client and server?
Is the application client-side?
Is there client-side activities involved in the entire application?
What is the authentication mechanism?
Does the application allow multiple login of a single user?
How does the application maintain a session?
How does the load balancer distribution the load?
What are the parameters been sent to the server?

Basically, this is actually gathering the technical details of the application unlike the initial Soliciting Requirements which comprises of the “big picture” information.

Without knowing how it works and you start clicking the links and buttons, you are bound for a failing load test at the script level. Also if you were intending to seek assistance from the paid support, chances is that they will ask you if you know the application well enough such as protocol used, client-side activities, and so on so forth.

This information should be consulted with the application developer as he/she is the one that developed it and will be the most knowledgeable amount the performance test team.

However, take note that you may bump into an inexperienced developer of the application, which might end up providing wrong information and wasting everyone’s time and effort. My advice is to be sure of the technology you are working with before approaching them.

Another thing to note, from a consultant or vendor company perspective, I do recognize that the load tester may have trouble conveying such messages or questions to your client’s development team. The time for them to answer your questions might not be as fruitful as writing another two modules to them. But bare minimum is that you must be able to translate and convince them the importance for the need to know the application design.

Related Topics

General: Planning for Load Testing
General - Content Page

General: Planning for Load Testing - Soliciting Requirements

2007-07-21T00:53:00.000+08:00

Prior to this stage, I always give a “sales talk” of what we do for a living and also to get them excited (if they are not). Gathering and soliciting accurate requirements will save you lots of unnecessary work.

Be sure to get hold of the requirements to monitor such as acceptable Transaction Response Time or server utilization. Clients need to be clear of what they want before you can proceed to achieve (unless is you who proposes the benchmark). For example, if CIO wants the response time for a search to be fewer 10 seconds will be a good metric to achieve. This is also the exit criteria as per se for the load test project.

They may also like to know the utilization of the servers under load and not really concerned with the response time. Usually, these are applications that are already problematic and it’s what they are hiring you to dig the cause of the bottleneck.

Always get an overview of the architecture which includes all components that inhabitant it. You can leave out the number of switches or routers if they do not provide much usage except routing and separating networks. It is important to get information like number of servers and what is housed in it, load balancers, firewall and databases. This will aid you in discussion and knowing what machines to be monitored.

Assist the clients in understanding which business process may raise performance issues or they see it’s criticality in the business competitive edge. With this, it can translate into the scripts that will be used in the load test.

Related Topics

General: Planning for Load Testing
General - Content Page

General: Planning for Load Testing

2007-07-19T11:01:00.000+08:00

After writing a couple of posts for the past few months, I would like to touch on the basic understanding of using LoadRunner in Performance Testing. This will provide an overview how you should facilitate the entire load testing process and be aware of the requirements that surfaces. If course, this is not limited to other load testing tools because the concepts are the same.

There are few areas to be aware or trained in order to have the load test proceed with minimal difficulties. I’ve broken them down into the following: namely, soliciting requirements, application design, protocols, monitor setup, monitoring, analyzing and recommendations.

Soliciting Requirements

Related Topics

General - Content Page

Understanding Memory: Virtual Addressing

2007-07-15T11:55:00.000+08:00

Virtual memory is a feature supported by most advanced processors. Hardware support for virtual memory includes hardware mechanism to map from logical (virtual) memory address that application programs reference to physical (real) memory hardware addresses.

When an executable program’s image file is first loaded into memory, the logical memory address range of the application is divided into fixed-size chunks called pages. These logical pages are them mapped to similar-sized physical pages that are resident in real memory.

This mapping is dynamic in that frequently referenced logical addresses tend to reside in physical memory (also know s RAM, real memory, or main memory), while infrequently referenced pages are relegated to paging files on secondary disk storage. The active subset of virtual memory pages associated with a single process’s address space currently resident in RAM is known as the process’s working set, because they are the active pages referenced by the programs as it executes.

Virtual addressing is designed to be transparent to application programs, allowing them to be written without regard to specific real memory limitations of this or that computer. Virtual addressing even makes it possible for an executing program to reference an address space that is larger than the amount of physical memory installed on a particular computer. Virtual addressing allows a programmer to exploit what looks like a virtually infinite-sized computer memory where each individual process can address up to 4 GB of virtual addresses. As of the above, performance issues still exists when the OS attempts to reference more memory locations than can actually fit inside real memory.

Virtual memory system works well because executing programs seldom require all the associated code and data areas they allocate to be resident in physical memory concurrently in order to run. With virtual memory, only the active pages associated with a program’s current working set remain resident in real memory. On the other hand, virtual memory systems can run very poorly when the working sets of active processes greatly exceed the amount of physical RAM that the computer contains. It is important to understand the logical and physical memory usage to diagnose performance problems arising from real memory being over-committed.

The above was extracted from the book, Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos.

Related Topics

Content Page - General
Page Fault Resolution
Page Fault Resolution (Illustration)
Performance Concerns
Virtual Memory Shortage Alerts
Available Bytes
LRU
System Working Set
Detecting Memory Leaks
Measuring Memory Utilization

General: Understanding Memory - Page Fault Resolution (Illustration)

2007-07-15T11:15:00.001+08:00

Illustration of Page Fault Resolution. To better understand how Page Fault occurs with respect to the illustration, refer to article, "General: Understanding Memory - Page Fault Resolution".

Fig 1: Page Fault Resolution

The sequence of Page Fault resolution as follows.

Thread attempts to reference page in memory.
Page not resident in real memory thus not found.
Hardware interrupt occur to resolve page fault.
ISR gains control to validate the referenced address.
Locate page on secondary storage.
Copy page into available free page in real memory.
Resume thread execution cycle.

The time taken for Page Fault Resolution thus equivalent from Step [3] to [6]. Having mentioned that, if memory is the performance bottleneck with the number of Page Faults increases over time, the application/system may experience delay due to the time needed to perform Page Fault Resolution pointed in Step [3] to [6].

Related Topics

Page Fault Resolution
Performance Concerns
Virtual Memory Shortage Alerts
Available Bytes
LRU
System Working Set
Detecting Memory Leaks
Measuring Memory Utilization

Basics: Network Bottlenecks

2007-07-14T23:18:00.000+08:00

At the network level, many things can affect performance. The bandwidth (the amount of data that can be carried by the network) tends to be the first culprit checked. Assuming you have determined that bad performance is attributable to the network component of an application, there is more likely cause of bad network performance than network bandwidth. The most likely cause of bad network performance is the application itself and how it is handling distributed data and functionality.

The overall speed of a particular network connection is limited by the slowest link in the connection chain and the length of the chain. Identifying the slowest link is difficult and may not even be consistent: it can vary at different times of the day or for different communication paths. A network communication path lead from an application through a TCP/IP stack (which adds various layers of headers, possibly encrypting and compressing data as well), then through the hardware interface, through a modem, over a phone line, through another modem, over to a service provider’s router, through many heavily congested data lines of various carrying capacities and multiple routers with different maximum throughputs and configurations, to a machine at the other end with its own hard interface, TCP/IP stack and application. A typical web download route is just like this. In addition, there are dropped packets, acknowledgements, retries, bus contention, and so on.

Because so many possibilities causes of bad network performance are external to an application, one option you can consider including in an application is a network speed testing facility that reports to the user. This should test the speed of data transfer from the machine to various destinations: to itself, to another machine on the local network, to the Internet Service Provider, to the target server across the network, and to any other destinations appropriate. This type of diagnostics report can tell users that they are obtaining bad performance from something other than your application. If you feel that the performance of your application is limited by the actual network communication speed, and not by other (application) factors, this facility will report the maximum possible speeds to your user.

Latency

Latency is different from the load-carrying capacity (bandwidth) of a network. Bandwidth refers to how much data can be sent down the communication channel for a given period of time and is limited by the link in the communication chain that has the lowest bandwidth. The latency is the amount of time a particular data packet takes to get from one end of the communication channel to the other. Bandwidth tells you the limits within which your application can operate before the performance become affected by the volume of data being transmitted. Latency often affects the user’s view of the performance even when bandwidth isn’t a problem.

In most cases, especially Internet traffic, latency is an important concern. You can determine the basic round-trip time for a data packets from any two machines using the ping utility. This utility provides a measure of the time it takes a packet of data to reach another machine and be returned. However, the time measure is for a basic underlying protocol (ICMP packet) to travel between the machines. If the communication channel is congested and the overlying protocol requires re-transmissions (often the case for Internet traffic), one transmission at the application level can actually be equivalent to many round trips.

It is important to be aware of these limitations. It is often possible to tune the application to minimize the number of transfers by packing data together, caching and redesigning the distributed application protocol to aim for a less conversational mode of operation. At the network level, you need to monitor the transmission statistics (using the ping and netstat utilities and packet sniffers) and consider tuning any network parameters that you have access to in order to reduce re-transmissions.

TCP/IP Stacks

The TCP/IP stack is the section of code that is responsible for translating each application-level network request (send, receive, connect, etc.) through the transport layers down to the wire and back up to the application at the other end of the connection. Because the stacks are usually delivered with the operation system and performance-tested before delivery (since a slow network connection on an otherwise fast machine and fast network is pretty obvious), it is unlikely that the TCP/IP stack itself is a performance problem.

In addition to the stack itself, stacks include several tunable parameters. Most of these parameters deal with transmission details beyond the scope of the book. One parameter worth mentioning is the maximum packet size. When your application sends data, the underlying protocol breaks the data into packets that are transmitted. There is an optimal size for packets transmitted over a particular communication channel, and the packet size actually used by the stack is compromise. Smaller packets are less likely to be dropped, but they introduced more overhead, as data probably has to be broken up into more packets with more header overhead.

If your communication takes place over a particular set of endpoints, you may want to alter the packet sizes. For a LAN segment with no router involved, the packets can be big (e.g. 8KB). For a LAN with routers, you probably want to set the maximum packet size to the size the routers allow to pass unbroken. (Routers can break up the packets into smaller ones; 1500 bytes is the typical maximum packet size and the standard for the Ethernet. The maximum packet size is configurable by the router’s network administrator.) If your application is likely to be sending data over the Internet and you cannot guarantee the route and quality of routers it will pass through, 500 bytes per packet is likely to be optimal.

Network Bottlenecks

Other causes of slow network I/O can be attributed directly to the load or configuration of the network. For example, a LAN may become congested when many machines are simultaneously trying to communicate over the network. The potential throughput of the network could handle the load, but the algorithms to provide communication channels slow the network, resulting in a lower maximum throughput. A congested Ethernet network has an average throughput approximately one third the potential maximum throughputs. Congested networks have other problems, such as dropped network packets. If you are using TCP, the communication rate on a congested network is much slower as the protocol automatically resends the dropped packets. If you are using UDP, your application must resend multiple copies for each transfer. Dropping packets in this way is common for the Internet. For LANs, you need to coordinate closely with network administrators to alert them to the problem. For single machines connected by a service provider, suggesting improvements. The phone line to the service provider may be noisier than expected: if so, you also need to speak to the phone line provider. It is also worth checking with the service provider, who should have optimal configurations they can demonstrate.

Dropped packets and re-transmissions are a good indication of network congestion problems, and you should be on constant lookup for them. Dropped packets often occur when routers are overloaded and find it necessary to drop some of the packets being transmitted as the router’s buffer overflow. This means that the overlying protocol will request the packets to be resent. The netstat utility lists re-transmission and other statistics that can identify these sorts of problems. Re-transmissions may indicate that the maximum packet size is too large.

DNS Lookup

Looking up network address is an often-overlooked cause of bad network performance. When your application tries to connect to a network address such as foo.bar.somthing.org (e.g. downloading a webpage from http://foo.bar.something.org), your application first translates foo.bar.somthing.org into a four-byte network IP address such as 10.33.6.45. This is the actual address that the network understands and uses for routing network packets. The is this translation works is that your system is configured with some seldom-used files that can specify this translation, and a more frequently used Domain Name System (DNS) server that can dynamically provide you with the address from the given string. DBS translation works as follows:

The machine running the application sends the text string of the hostname (e.g. foo.bar.something.org) to the DNS server.
The DNS server checks its cache to find an IP address corresponding to that hostname. If the server does not find an entry in the cache, it asks its own DNS server (usually further up the Internet domain-name hierarchy) until ultimately the name is resolved. (This may be by components of the name being resolved, e.g. first .org, then something.org, etc, each time asking another machine as the search request is successively resolved.) This resolved IP address is added to the DBS server’s cache.
The IP address s returned to the original machine running the application.
The application uses the IP address to connect to the desired destination.

The address lookup does not need to be repeated once a connection is established, but any other connections (within the same session of the application or in other session s at the same time and later) need to repeat the lookup procedure to start another connection.

You can improve this situation by running a DNS server locally on the machine, or on a local server if the application uses a LAN. A DNS server can be run as a “caching-only” server that resets its cache each time the machine is rebooted. There would be little point in doing this if the machine used only one or two connections per hostname between successive reboots. For more frequent connections, a local DNS server can provide a noticeable speedup to connections. Nslookup is useful for investigating how a particular system does translations.

Related Topic

Content Page - General

Tools: Monitoring Unix Resource Script ver 1.0

2007-07-11T09:23:00.001+08:00

Due to the past experiences working in my organization, having the problem of not able to monitor the Unix system resources due to whatever reasons (rstatd configuration, dynamic ports, security reasons or lazy administrators) had always been bothering us.

Our team came up with the simple initiative to monitor using the same concept as perfmon. We requested (outsourced) one of our interims to search for unix monitoring scripts and modify it to collect data periodically. And here we are, special thanks to the interim, we finally got the first-cut of the scripts.

Copy and paste the below script starting from here ==================================================================

#!/bin/bash
# (C) 2006 Mark Boddington, http://www.badpenguin.co.uk/

# (C) 2007 Modified by Hwee Seong, http://loadrunnertnt.blogspot.com
# Licensed under the GNU GPL Version 2.
# ***** Configuration *****
# set LOG to the directory you want to write the performance data to.
# set HDD to the number of had disks in your machine.
# NOTE : Do not change the variable LIMIT
LOG=/home/Admin/report
HDD=2
LIMIT=0
genStat()
{
test=$1
SLEEP=$2
filename=$(date +%Y%m%d%H%M%S)
for((num=0;num> ${LOG}/vm.${filename}.log &
sleep $SLEEP
done
if [[ LIMIT -eq test ]]
then
#VM csv
echo Date,Time,Procs in Run Queue,Procs Blocked,Virtual Memory Used,Idle Memory,Buff Memory Used,Cache Memory Used,Swap In,Swap Out,Blocks Received,Blocks Sent,System Interrupt,Context Switches/sec,Cpu User Time,Cpu System Time,Cpu Idle Time,Cpu Waiting Time> ${LOG}/vm.${filename}.csv
cat ${LOG}/vm.${filename}.log awk '{for(l=1;l<17;l++)>> ${LOG}/vm.${filename}.csv
fi
}
case $1 in
run)
genStat $2 $3
;;
*)
echo -e ":::: Usage ::::"
echo -e "$0 run 5 10 : Collect stats for 5 times with 10 seconds interval"
echo -e "Note : Change the first parameter for the number of stats"
echo -e " Change the second parameter for the number of seconds between each stats"
;;
esac

==================================================================

The original script was developed by Mark Boddington and could be obtained from http://www.badpenguin.co.uk

The modified script allows the interval between each collection and the number of times the collection is made. However, this script lacks the ability to set the output path and it is created based on vmstat. Also, you will have to estimate the completion time for the scenario to complete.

To use the script, copy the quoted content (above) and name it MonitorPerformance.sh. Then in the directory that the script exists, type the following:

\MonitorPerformance.sh 5 10

Where 5 is the number of times the collection is made and 10 is the interval between each collection in seconds.

Once the collection is complete, import the file in analysis. Our settings in Analysis are as follows.

This is a minimal start for us and we intend to further enhance the script to allow the following (provided we can squeeze some time out):

1. Changing of the output path

2. Include iostat and netstat.

Please feel free to use the script and we welcome suggestions, modifications and enhancement to the scripts to make it a better tool.

(simple tools making lifes easier)

Tools: Determine the Windows patches installed using Windows 2003 Resource Kit

Scripts: Starting a new transaction during iterations

2007-07-09T13:45:00.000+08:00

Recently one of my load testing projects required us to monitor the time taken to upload a file into the application. This sounds simple and straight forward with the implementation of the lr_start_transaction and the lr_end_transaction. However, the catch is that the action block be iterated 10 times and within the 10 iterations, one of the iteration is to upload a different file of different size (thus causing different response time), which they want to measure.

What we did was implement a if-then-else to check the file and perform a lr_start_transaction when the script is uploading the different file and end the transaction with the lr_end_transaction when the submission completes. The snippet of the code is as followed:

if (!strcmp(lr_eval_string("{file}"), lr_eval_string("{set_file}")))

{

lr_start_transaction("tx_check_upload_different_file");

}

else

{

lr_start_trasaction("tx_tx_check_same_upload_file");

}

...

\\other codes

...

...
if (!strcmp(lr_eval_string("{file}"),lr_eval_string("{set_file}")))

{

lr_end_transaction("tx_check_upload_different_file",LR_AUTO);

}

else

{

lr_end_trasaction("tx_check_same_upload_file",LR_AUTO);

}

The code detects the if the uploaded file is different and will start tracking the timing for the upload transaction. Hope it's useful to you.

Related Topics

Scripts: Duplicating Files on the Fly

Scripts: Step Download Timeout

Scripts: VIEWSTATE

Scripts: Auto or Manual Correlation?

Scripts: Remove Think Time

Scripts: Set Debug Mode in Script

Scripts: Replay Failure – Use Full Extended Log!

Basics: Memory Bottlenecks

2007-07-08T16:02:00.000+08:00

Maintaining watch directly on the system memory (RAM) is not usually that helpful in identifying performance problems. A better indication that memory might be affecting performance can be gained by watching for paging of data from memory to the swap files. Most current OS have a virtual memory that is made up of the actual (real) system memory using RAM chips, and one or more swap files on the system disks. Processes that are currently running are operating in real memory. The OS can take pages from any of the processes currently in real memory and swap them out to disk. This is known as paging. Paging leaves free space in real memory to allocate to other processes that need to bring in a page from disk.

Obviously, if all the processes currently running can fit into real memory, there is no need for the system to swap out any pages. However, if there are too many processes to fit into real memory, paging allows the system to free up system memory to run more processes. Paging affects system performance in many ways. One obvious way is that if a process has had some pages moved to disk and the process becomes run-able, the OS has to pull back the pages from dusk before that process can be run. This leads to delays in performance. In addition, both CPU and the disk I/O spend time doing the paging, reducing available processing power and increasing the load on the disks. This cascading effect involving both the CPU and I/O can degrade the performance of the whole system in such a way that it maybe difficult to even recognize that paging is the problem. The extreme version of too much paging is thrashing, in which the system is spending so much time moving pages around that it fails to perform any other significant work. (The next step is likely to be a system crash).

As with run-able queues (see CPU section), a little paging of the system does not affect the performance enough to cause concern. In fact, some paging can be considered good. It indicated that the system’s memory resources are fully utilized. But at the point where paging becomes a significant overhead, the system is overloaded.

Monitoring paging is relatively easy. On UNIX, the utilities vmstat and iostat provide details as to the level of paging, disk activity and memory levels. On Windows, the performance monitor has categories to show these details, as well as being able to monitor the system swap files.

If there is more paging than is optimal, the system’s RAM is insufficient or processes are too big. To improve this situation, you need to reduce the memory being used by reducing the number of processes or the memory utilization of some processes. Alternatively, you can add RAM. Assuming that it is your application that is causing the paging (Otherwise, either the system needs an upgrade, or someone else’s processes may also have to be tuned), you need to reduce the memory resources you are using.

When the problem is caused by a combination of your application and others, you can partially address the situation by using process priorities (see the CPU “section”). The equivalent to priority levels for memory usage is an all-or-nothing option, where you can lock process in memory. This option is not available on all systems and is more often applied to shared memory than to processes, but nevertheless, it is useful to know. If this option is applied, the process is locked into real memory and is not paged out at all. You need to be aware that using this option reduces the amount of RAM available to all other processes, which can make overall system performance worse. Any deterioration in system performance is likely occurring at heavy system load, so make sure you extrapolate the effect of reducing the system memory in this way.

Related Topic

Content Page - General

Products: SilkPerformer (a comparison with LoadRunner)

2007-07-06T16:21:00.000+08:00

SilkPerformer is a load testing tool from Borland, originally Segue. SilkPerformer have almost all features that LoadRunner have. Therefore, you may like to read about the product LoadRunner here before proceeding further on this article. The SilkPerformer suite consists of the Workbench, Silk TrueLog and Silk Performance Explorer. The entire load test implementation is the same where the load generators have to be installed at the machines. For the sales talk by the vendor, click here.

What SilkPerformer offer is it handles load test in project approach. Using this approach, each load test is stored as a project with various settings of profiles and workloads. This is advantageous when managing the load test in a customer or project perspective where you can track the projects accordingly. Furthermore, within a project, for every load test run, a result directory for each run is created. This is useful in tracking the number of runs that were conducted. In this way, there is a structural approach in managing the tests that were conducted. Both features mentioned above are not available in LoadRunner.

SilkPerformer have another cool feature, which is resource management of the load generator. By defining a pool of load generators, the load tester is able to have an overview of the resources that is participating in the load test. This is almost similar to the Resource Pool in Performance Center. This is not available in LoadRunner.

During the execution, SilkPerformer allows monitoring of the resources in terms of CPU usage in the Load Generator. This is beneficial when differing problem in resources on the Load Generator or the SUT itself. Furthermore, feature allows capacity planning for load generators. SilkPerformer also allows monitoring on the progress of the vusers and load test in terms of percentage that displays the progress in percentage of the vusers. This is beneficial in estimating the duration of the load test and forecast the end time of the load test.

While performing the evaluation, I found that the parameterization feature offered in SilkPerformer is not as user-friendly as compared to LoadRunner. The manipulation of the parameterization is not as straightforward such as defining the parameter and reusing the parameter unlike LoadRunner where a replace function can easily achieve the parameterization.
For iteration, SilkPerformer have to define the iteration number in the script. For pacing a LoadRunner feature, is not offered in the SilkPerformer.

SilkPerformer provides Rendezvous feature in terms of serialisation and synchronisation. This is useful if a certain load model needs to be emulated during the test (better than LoadRunner). However, the downside is it requires inclusion of the API which is not as convenient as LoadRunner in implementing a Rendezvous point.

For replay, the Silk TrueLog is a powerful tool in analysing the replay as compared to LoadRunner Runtime Viewer. If offers replay of the screens and providing views of the data in and out from the client and server.

SilkPerformer allows monitoring similar to LoadRunner. However, LoadRunner provides more features in the monitoring perspective. SilkPerformer requires additional logging for non-default monitors unlike LoadRunner, which automatically logs the data when the counter is been added into the monitor list. Also, SilkPerformer is restricted to a defined set of custom monitors namely JMX, SNMP, Perfmon and Rexec unlike LoadRunner, which provides more monitors.

SilkPerformer utilises Silk Performance Explorer for analysing. However, it’s capability to merge, manipulate, handle external data is not as flexible and robust as LoadRunner Analysis. The analysing feature is an important component used by the Performance Team to determine problem causes.

SilkPerformer have the similar license mechanism as LoadRunner, which is bounded by a single host. However it features an additional mechanism that allows the option to "check-out" license to the installed application. This is useful in an organisation context in managing our performance test in different environments.

Related Topics