Deep Diagnostics Explained

Here are notes on HP (Mercury) Deep Diagnostics from the inside out. Tidbits here were culled from manuals and work experience. This is a companion page to my pages on Mercury Diagnostics Products, Performance Monitoring, Tuning, and Testing using LoadRunner and other products.

Site Map  
About this site  

Diagnostic Components

Deep Diagnostics captures transaction data such as page hits, JSP object requests, EJB and JDBC calls, and database SQL requests over time so that latencies and throughput can be correlated across multiple tiers of the application.

Instead of a mediator, Mercury's Deep Diagnostics ("DD") features use DD Agents installed on three machines:

• inside the application server under test
• a DD Server machine
• a DD Console machine (the "Commander")

Deep Diagnostics was renamed from the "OptiBench" product after Mercury acquired it from the Performant Corporation. This is why the default log filename used by Deep Diagnostics Instrumented and post-processing Java code is named performant.log. An abbreviation of this name is used to name the pfmt.logging.logfile property.

Mercury acquired the "Total Trace" technology to fold into Mercury's own Diagnostics product. So DD was not significantly upgraded from v3.6.4.1 dated 8/5/2005, which does not diagnose multi-CPU machines.

Tree

Each node in the Tree View panel represents an Object. Switch from one node to another by using the mouse or the keyboard: Shift+Up Arrow moves to the current node's parent. If the node has no parent, then moves to the next node up in the tree. For example, if a Workload Capture Action Plan is selected, and you press Shift+Up Arrow, the run plan would be selected. If a folder is selected, then the folder above it would be selected, or the project node would be selected if there are no more folders. Shift+Down Arrow Moves to the current node's first child. If the node has no children, then moves to the next node down in the tree. Shift+Left Arrow moves to the next node up in the tree, regardless of parent/child relationships. Shift+Right Arrow moves to the next node down in the tree, regardless of parent/ child relationships.

Deep Diagnostics organizes its repository into this structure:

Folder Projects (described with text of goals, procedure, results, and notes)

Folder
Experiment (runs), each with its own capture plan of:
Tier Capture Plan actions
Metric Capture Plan actions

Reports
Profiles and call graphs

Capture Plans

Probes capture performance values that typically include CPU utilization, main memory consumption, disk I/O requests, and network traffic data throughput.

Experiments reference global values within this Server Status classsifications:

	Host Definitions (servers with Deep Diagnostics Agents installed)
	HTTP Server Profiles (paths to the capture module for each server)
	Metric Definitions (of system metrics collectible from hosts)
	Tier Definitions (classes and methods that can be captured at each tier)
	Instrumentation Plans (in the definition of each tier)
	Application Definitions (configuration parameters such as paths to source code for drill down)

The above are listed in the order they are populated with values.

Deep Diagnostics runs use command files written in Jython (Python within Java) in the scripts folder. These scripts reference properties in the /shared/etc folder.

Analysis of Deep Diagnstic run results to identify concurrency problems:

• Organize calls by request type, drag and drop metrics for graphical comparison, and adjust scale to clarify relationships.
• Isolate specific calls by time, object type, call type, and argument passed.
• graph memory used over time,
• correlate calls, correlations of performance,
• identify concurrency problems
• diagram call paths across multiple object tiers,
• link tables to show instance-level details.
• Export data for additional analysis and reporting in external business intelligence tools such as SAS.

DD uses four native libraries: ptimer, pfilelog, ploglib, putil.

For system components in which the source code is not available and modification of the object code is inappropriate or violates a software license agreement, Mercury Professional Services Group, can use the interposition technique that substitutes a system component with a pass-through component that records calls made to it.

Deep Diagnostics Profiles and Call Graphs

Unlike static call graphs created by IDEs, DD call graphs are created dynamically from actual execution paths.

• Local calls are represented by rectangles.
• Remote RMI/IIOP calls are representated by turquoise filled, gem-shaped polygons.
• Frequently called methods are shown with thick edges (or whatever thickness you choose in the Throughput Coloring lens)
• High latency calls are colored red (or whatever color you choose in the Latency Coloring lens)

In Profile graphs:

• Rectangles directly beneath a given parent rectangle represent calls invoked within the scope of the parent.