JMeter Load Testing

This is an insider's explanation of JMeter, the popular open-source Java-based tool for load testing. This succintly describes (minus the sales hype) how to install, configure, and use JMeter on Windows. But this does more than take readers through the motions. Noted Test Architect Wilson Mar explains the internals of how JMeter works, one step at a time.

See related pages on load testing tools, Testing

Topics this page:

Site Map  
About this site  

Why JMeter?

Like other software used to conduct load testing,
JMeter from the Apache Jakarta project (first available 12/03), consists of a Master system (the Jmeter GUI) which controls remote slave systems (running jmeter-server) which concurrently imposes load on a target server, network or object under test by emulating traffic to and from client software.

Unlike Selenium, which controls a browser (and changes Widows Registry to control IE), JMeter (like LoadRunner) emulates what goes over the wire in and out between a browser and the server under test.

• JMeter does not process the Javascript applets embedded in HTML pages (such as dynamic AJAX calls).

Unlike OpenSTA, JMeter is able to record traffic in native WinInet protocol as well as HTTP protocol, which may explain why OpenSTA response times stats vary from LoadRunner and JMeter for the same app/machine

Unlike LoadRunner, JMeter is open-source and thus free to download without licensing headaches. JMeter's inventor, Mike Stover, remains actively involved in the JMeter Mailing List.

Although JMeter presents graphs of run results (end-to-end performance under loads over time), LoadRunner, provides wider flexiblity in formatting and more fine-grained analysis.

See usermanual jmeter distributed testing step_by_step

JMeter Architecture

The JMeter GUI (ApacheJmeter.jar in the bin folder) is a multi-threaded Java class running Java Swing interfaces. It can be invoked by jmeter.bat

It communicates with multiple remote injector Java RMIRegistry services.

Remote servers by default listens to port 1099.

Each Thread Group simulates what LoadRunner calls an individual virtual user.
Each thread is a unit of work that can be executed simultaneously or sequentially.
To each thread group can be added Logic Controllers and Elements.

A Test Plan is a container for elements which specifies the parameters for test runs.

JMeter FAQ Wiki at Apache.org

DevX article

Build Quality Apps with JMeter 5 page article (dated August 2002) by Peter Varhol

Download and Installation

Identify the folder path where you will be downloading JMeter.
Example: C:\projects\Perf\
If you use the Eclipse IDE:
Example: ...\eclipse\workspace\

Download from the middle of this page a JMeter Binary file such as 2.3.2.zip for Windows machines (16.6 MB created 6/10/2008).

• Unzip folder jakarta-jmeter-2.3.2 into the folder designated.

For reference later, after unzipping, we copy the zip file inside the expanded folder.

• We also download the JMeter Source file such as 2.3.2.zip for Windows machines (5.93 MB created 6/10/2008).

• Specify the path in a new environment variable JMETER_HOME.

Verify that the downloaded file contains all the bytes.
Since Microsoft does not provide an MD5 verification program with its operating system, you need to download one from fourmilab or pc-tools.net or slavasoft.

Click on the [md5] link associated with the JMeter file you downloaded. Highlight and copy what looks like this:

314099349e1b7edc5de7005d9311c5ad *jakarta-jmeter-2.3.2.zip

and paste in that ???

Folders and Environment Variables

JMeter is a Java application. But JMeter ignores the environment variables CLASSPATH referenced by other Java programs.

• Determine the file path where java.exe and javaw.exe are installed. A common value is:
C:\Program Files\Java\jre1.6.0_07
• Specify that path value when creating system environment variable JAVA_HOME

Files from Jakarta are organized into folders common to most Unix distributions:

• docs\api contains static HTML generated from Jmeter java code packages starting from file \docs\api\overview-summary.html. If you're low on disk space, this can be deleted since you can view them online here.
• printable_docs\usermanual contains a User Manual. To save disk space, these files can be deleted if you refer to them online here.
• extras contains sample jmeter-results-reports (in XML) and
  ant-jmeter-1.0.9.jar. To run JMeter test plan programatically using Ant, use this ant task script
• lib contains jar (java archive) files. JMeter automatically finds JMeter components in jars within the
  lib\ext directory (containing by default ApacheJMeter jar files). So jars (not zips) such as JDBC and new custom JMeter components should go there.

lib\junit contains a test.jar
lib\api contains geronimo jars if src was installed.
lib\doc contains a velocity jar if src was installed.
• xdocs folder exists if src was installed.
• src folder exists if src was installed.
• test folder exists if src was installed.
• printable_docs\Extending folder contains How to Write a plugin for Jmeter to extend Jmeter
• printable_docs\demos contains jmx files that specify demo test plans (in XML format). Custom test plans can be recorded via HTTP and HTTPS using badboy ($45 after 15 days).
• bin contains .cmd and .bat Windows command files to invoke JMeter.
  bin also contains configuration text file jmeter.properties which stores settings such as IP addresses of remote_hosts setup to run JMeter.
  To enable JMeter to locate these files on Windows machines, create system environment variable JMETER_BIN to specify a path to the Jmeter bin folder. CAUTION: The path value must end in \.
• bin\testfiles is present if src was unzipped.

jmeter.bat Invocation Commands

The JMeter GUI is invoked by specifying the jar file which java.exe JRE invokes from within the JMETER_BIN folder:

-jar "%JMETER_BIN%ApacheJMeter.jar"

This statement is within bin file jmeter.bat, which creates environment variable ARGS containing all parameters jmeter uses, which includes the PERM variable to specify JVM permanent space allocation.

If needed, system environment variable JVM_ARGS can be defined to specify additional java options needed by custom Java extensions, e.g. -Dprop=val

Additionally, run parameters read by the JMeter java program can be specified:

-v or --version displays the version.

-n specifies JMeter to run in non-gui (command line) mode.

-t precedes the name of the .jmx file containing the Test Plan.

-l precedes the name of the .jtl test log file containing sample results.

-r or --runremote runs all remote servers specified in jmeter.properties The default is the localhost:

remote_hosts=127.0.0.1

-R overrides jmeter.properties with a list of server IPs separated by commas.

If the site to be tested is behind a corporate proxy, add:

-H my.proxy.server -P 8000 -u username -a password

In the bin folder .cmd files jmeter-n.cmd and jmeter-n-r.com have these parameters pre-specified. The "-n" and "-r" in the name reflect the JMeter run parameters within each cmd file.

A jmx test file can be executed by dragging it from one Windows Explorer and dropping it onto the cmd file shown on another Windows Explorer window.

If you see a black screen flash and disappear, a pause command is needed.

Alternately, for the convenience of one-click invocation, we create a custom cmd file containing the jmx file to be executed. For example, custom invocation file SimpleTestPlan.cmd would contain:

jmeter.bat -t%JMETER_HOME%\printable_docs\demos\SimpleTestPlan.jmx
-Ljmeter.engine=DEBUG -X
pause

Java run parameters have no space between the tag (such as "-t") and the value.

-LDEBUG is equivalent to -Ljmeter.engine=DEBUG

-G defines a JMeter property sent to all remote servers.
-Jx=2 defines a local JMeter property named "x" with value "2" referenced within test plans as variable ${__property(x)}. This is used to define the threads, rampup, and loops for each group.
-D defines a Java system property. This is needed to get around connection exceptions
-Djava.rmi.server.hostname=192.168.5.6

-X exits the server after running a single test. This overrides the properties value to exit after running a single test:

server.exitaftertest=true

A full list of the parameter can be obtained by:

jmeter -?

jmeter.properties

There are several reasons to edit the jmeter.properties file.

To exit after running a single test:

server.exitaftertest=true

See usermanual jmeter_proxy_step_by_step.pdf on setting up the proxy to record a HTTP session.

The jmeter.properties file contains references to other properties files:

• system.properties
• user.properties

Test Plan .jmx files

Sample test plans from the installer in folder \printable_docs\demos are:

AssertionTestPlan.jmx
AuthManagerTestPlan.jmx
BeanShellAssertion.bsh
ForEachTest2.jmx
forEachTestPlan.jmx
HeaderManagerTestPlan.jmx
InterleaveTestPlan2.jmx
InterleaveTestPlan.jmx
JMSPointToPoint.jmx
LoopTestPlan.jmx
OnceOnlyTestPlan.jmx
ProxyServerTestPlan.jmx
SimpleTestPlan.jmx of Jakarta users
URLRewritingExample.jmx

Within a .jmx file, the first line identifies it as being encoded in UTF-8. So it's better to open .jmx files using a text editor such as WordPad, which knows to save in that format. (rather than Notepad).

<jmeterTestPlan version="1.2" properties="2.1">
This second line specifies which version of the JMeter API to which the file was defined.

All content sections are nested within hashTree tags. I don't know why.

Values in the TestPlan section are pulled from properties files.

The number within ThreadGroup.start_time and end_time (1211836583000 in v3.2) is (when the trailing "000" is removed), the Unix Epoch time for "Mon, 26 May 2008 21:16:23 UTC".

YouTube Video:
jmeter getting started uses a Mac to create a HTTP Proxy.

Jörg Erdmenger's vimeo:
Apache JMeter - The basics tutorial on contextual settings.
Apache JMeter Getting Started Tutorial
Apache Jmeter with a simulated login

My Sample .jmx

Download and open this template file I start with when creating a JMeter Test Case (based on this wiki page).

.

Test Elements

Right click on a thread group to add elements, which are executed in this order within each controller:

1. Pre-Processors execute functions before requests are sent:

   HTML Link Parser, HTTP URL Re-writing Modifier, HTML Parameter Mask, HTTP User Parameter Modifier, User Parameters, Counter, BeanShell PreProcessor

2. Timers add delays between responses: Constant, Gaussian Random, Uniform Random, Constant Throughput, Synchronizing, BeanShell
3. Samplers (introduced with v2.1.2) are protocol specific. Protocols supported by JMeter samplers include:
   • HTTP Request
   • FTP Request
   • JMS (Java object request)
   • JDBC Request
   • LDAP Request Extended Ops
   • SOAP/XML-RPC Request
   • WebService (SOAP) Request
   • JNDI
   • POP Mail (not SMTP)
   • TCP

   The JUnit Sampler includes methods for "oneTimeSetUp" and "oneTimeTearDown". These should be thread-safe (not static methods).

4. Post-Processors execute functions after requests are sent.

   Regular Expression Extractor, XPath Extractor, Result Status Action Handler, Save Responses to a file, Generate Summary Results, BeanShell PostProcessor

5. Assertions determine if the application is returning the results expected: Response, Duration, Size, XML, BeanShell, MD5Hex, HTML, XPath, XML Schema
6. Listeners perform data collection, aggregation, and visualization (display):

   Sample Result Save Configuration, Graph Full Results, Graph Results, Spline Visualizer, Assertion Results, Aggregate Report, View Results in Table, Simple Data Writer, Monitor Results, Distribution Graph (alpha), Aggregate Graph, Mailer Visualizer, BeanShell Listener, Summary Report

   View Results Tree requires a large amount of memory and processing resources.

   • Logical controller elements tell JMeter when to send requests: Simple, Once Only, Loop, Random, Random Order, If, While, Switch, ForEach, Module, Throughput, Runtime, Interleave, Include, Transaction, Recording
   • Generative Controller elements communicate with applications.

"Parsing XML is CPU intensive. On a Pentium 4 2.4ghz cpu, 50 threads is the upper limit."

Data Parametization

JMeter Wiki page How do I parameterize my JMeter test cases? outlines Regular Expressions to define user id's and other variables.

Data files

JUnit Integration

JMeter scans jar files in jmeter/lib/junit (instead of /lib directory) for classes extending junit's TestCase class.

TestNG ???

Extending JMeter

JMeter's 9.8MB Java 1.4/AWT-based source can be altered to add plug-ins that reference custom objects such as those in custom JDBC apps.

Download the source.

Run Results Graphs

The JMeter GUI recognizes key combinations, which are faster than menu:

To stop threads after current work is done: Ctrl + comma
To stop threads immediately: Ctrl + period

Custom Monitoring of JMeter

JMeter collects its own metrics from servers under test (such as CPU utilization), so if servers already have monitors in place, it's better to run JMeter without its normal monitors, and have the monitoring software monitor how many thread groups (vusers) is running.

Here's how to instrument JMeter for the Intersperse console.

1. Discover jmeter mbeans using the Intersperse JMX Remote Agent.

2. Open the Jmeter.bat file and add a JAVA_OPTIONS environment variable that uses "-D" to specify several Java variables to JMeter at start-up:
   set JAVA_OPTIONS=%JAVA_OPTIONS%
   -Dcom.sun.management.jmxremote
   -Dcom.sun.management.jmxremote.port=8004
   -Dcom.sun.management.jmxremote.authenticate=false
   -Dcom.sun.management.jmxremote.ssl=false

3. After adding %JAVA_OPTIONS% in Jmeter.bat file, the command to invoke Java should look like this:
   %JM_START% %JM_LAUNCH% %JVM_ARGS% %ARGS% %JAVA_OPTIONS% -jar ApacheJMeter.jar %JMETER_CMD_LINE_ARGS%

4. Install the Intersperse JMXRemoteAgent into a separate directory on the JMeter machine:
   Launch the Intersperse Installer. Choose "Custom Install" and only select "JMXRemoteAgent".
5. Invoke the JMX Remote Agent webconsole and add information about the JMeter server. This is done instead of manually editing the JMXRemoteAgent install directory file intersperse.jmxremoteagent.jsr160.servers within .... JMXRemoteAgent\config\JMXRemoteAgent\ where "host" is the machine hosting JMeter:

   remote.1.server.url=service\:jmx\:rmi\:///jndi/rmi\://host\:8004/jmxrmi
   remote.1.server.name=JMeter

   Note: The "\:" escape characters are automatically converted.

6. Save the file and exit.
7. Start the Jmeter application using jmeter.bat.
8. Start the Intesperse JMX Remote Agent.
9. On the Intersperse Console, view the Jmeter Mbeans Thread count, by navigating to "Threading Mbean" along with other Mbean "OS Monitoring" (such as cpu usage and memory usage).

Issues

JMeter is not "object oriented" in that it has a flat name and value pairs.

FunkLoad test is Python based. webunit.