Performance Project Planning

This companion to my Sample Load Testing Report describes several "best practices" and tools for load testing and performance engineering — activities for the capacity management of IT Service Management (ITSM).

“If you can't describe what you are doing as a process, you don't know what you're doing” —W. Edwards Deming

Site Map  
About this site  

How to Ensure Performance

There are two aspects to ensuring performance:

• The performance of the system under test, which is described in the sample performance profile report
• The process for conducting performance testing, which is described below.

Here is a consolidated list of performance engineering phases, tasks, and deliverables based on the ITIL and my own experience.

• Ideally, all this occurs AFTER considering the organization's quality maturity at ensuring smooth team development by clarifying participants' roles and risk-adjusted milestone schedules toward delivering a flow of artifacts created by achieving project process objectives using provisions needed for load-testing

1. Define [Identify] (internal and external) customer concerns issues formally as qualitative requirements and quantitative performance criteria with numerical Targets supporting service levels at optimal TCO.

   Study the Stystem/Application Under Test to specify the Test Architecture and Test Behavior. Define Test Timing Constraints. Prepare Test Data.

2. Measure (quantify/determine) current/existing Actual Load Patterns in response to Actual Usage Patterns by designing experiments and constructing and validating LoadRunner emulation script coding and Test Scenario parameters for the types of performance defects of concern.
3. Analyze the root cause(s) of bottlenecks (defects) revealed on reports and graphs created from results of Diagnosing load test runs while conducting performance tuning and exploring Various Configurations for Scalability in an analytical model

   Conclusions are published along with recommendations for management inquiries and action.

4. Improve [Optimize] the process by eliminating the root causes of botlenecks (defects) by implementing tuning changes possible to application code and configurations
5. Control [Verify] future process performance by monitoring production performance in Real Time so that Off-estimate Alerts are issued to perhaps trigger planned Upgrade trigger thresholds previously defined in Predicted Performance Profiles for anticipated loads

The approach above was drawn from several capacity management frameworks:

In the electronics industry:

Near the end of the Prototyping stage, after engineers create actual working samples of the product they plan to produce, Engineering Verification Testing (EVT) uses prototypes to verify that the design meets pre-determined specifications and design goals. This is done to validate the design as is, or identify areas that need to be modified.

After prototyping, and after the product goes though the Design Refinement cycle when engineers revise and improve the design to meet performance and design requirements and specifications, objective, comprehensive Design Verification Testing (DVT) is performed to verify all product specifications, interface standards, OEM requirements, and diagnostic commands.

Process (or Pilot) Verification Test (PVT) is a subset of Design Verification Tests (DVT) performed on pre-production or production units to Verify that the design has been correctly implemented into production.

The MOF (Microsoft Operations Framework) defines this circular process flow of capacity management activities:

1. Trend Analysis
2. Modeling
3. Optimization
4. Change Initiation
5. Monitoring

Oracle's Expert Services' Architecture Performance Capacity Scope & Assessment consulting uses these phases and deliverables:

Phase	Deliverable
1. Define Assessment Criteria	A. Assessment Scope B. Business and Functional Needs
2. Define High Level Requirements	C. High Level Requirements Matrix
3. Document System Baseline	D. Current Architecture Diagram (Baseline) Report
4. Generate Findings, Recommendations, and Conceptual Architecture	E. Conceptual Architecture Diagram F. Assessment Report G. Final Presentation

Software Performance Engineering (SPE)

Smith's Software Performance Engineering (SPE) approach begins with these detailed steps:
1. Assess Performance Risks such as antipatterns (recurring causes of problems). A flood of simultaneous requests is called "The Slashdot Effect" because the many readers of www.slashdot.org visit a site at the same time after it is mentioned on the online magazine.
2. Identify critical Use Cases
3. Select key performance workload scenarios (sequence diagrams detailing key use cases)
4. Establish performance objectives under specific workload intensities (such as best and worst case response times).
5. Define performance model(s): Queueing network models (QNM) use Information Processing Graph notation to illustrate the locus of execution among a sequence of queues. Execution Graphs illustrate the probability and frequency of execution of processing steps relevant to performance analysis.
6. Construct performance model(s)
7. Add software resource requirements (e.g., messages sent, database accesses, etc.)
8. Add computer resource requirements (e.g., CPU instruction utilization, disk I/O throughput, network connections, screen draws)
9. Evaluate performance model(s)

Anti-Patterns and Critical Success Factors

Why go through all the trouble above? They were developed to avoid Anti-Patterns — the chronic conditions (patterns) that keep projects from being efficient and effective.

Here's what can go wrong during a load testing project and the Critical Success Factors (CSFs) which ensure projects keep moving forward.

Anti-Pattern Symptions & Conditions	Critical Success Factor (CSF)
A. Runs are conducted with no agreement on their purpose.	Define requirements for the application.
	Define for the measurement project expected benefits, cost analysis, and justification.
	Define Objectives for each run is defined.
B. Much discussion about what to do next. Frustration with intermediate steps to obtain results.	Pre-agreement on deliverables from distinct project phases
C. Provisions for testing are not available	Accounting and Accountability for each role
D. Runs are conducted with scipts that are not ready/mature	Understanding of script maturity
E. People make mistakes after working 16 hours a day.	Planning and implementing an appropriate level of resources to match business needs
F. People getting into trouble with nothing productive to do.
G. Actual production usage is different than anticipated.	Availability of accurate and timely business forecasts
	Possible impacts to performance are identified and measured.
H. Changes by others are discovered as the reason for anomalies in run results — after many hours are wasted in debugging.	Communication and interaction with other service management processes
I. Tests need to be rerun because actual hardware used in production is different than hardware tested.	Understanding of current and future technologies (for each application in the Service Catalog, a mapping of infrastructure dependencies).

Test Policies

The test policy is the document which describes an organization's philosophy towards the testing (or quality assurance) of software. The test policy is complementary to, or a component of, the organization's Quality Policy, which describes the basic views and aims of a company, regarding quality, as pursued by management.

The Test Handbook is a framework document which describes the test steps to be excuted to address risks which should be covered by software testing. and the test activities to be carried out.

Risk is a combination of the possibility of the occurrence of a problem and the resulting effect of that problem.

The test concept describes the test steps and test activities to be executed for a particular project.

The test step plan details the procedure for a test step and describes implementation of the test concept for a particular test step.

Forms/Types of Performance Testing/Engineering

"Approaches to Performance Testing" by Matt Maccaux

Accomplishments

A. Speed Tests
(for Responsiveness)

conclusions

During speed testing, the user response time (latency) of each user action is measured.

The script for each action will look for some text on each resulting page to confirm that the intended result appears as designed.

Since speed testing is usually the first performance test to be performed, issues from installation and configuration are identified during this step.

Because this form of performance testing is performed for a single user (under no other load), this form of testing exposes issues with the adequacy of CPU, disk I/O access and data transfer speeds, and database access optimizations.

The performance speed profile of an application obtained during speed testing include the time to manually start-up and stop the application on its servers.

1. Identified the business processes under test.
2. Documented each user action to be measured.
3. Documented production installation configuration instructions and settings.
4. Quantified the start-up, shut-down, and user GUI transaction response (latency) times when the system is servicing only a single user at a time (under no other load) in order to determine whether they are acceptable.
5. Ensured CPU, disk access, data transfer speeds, and database access optimizations are adequate.

B. ContentionTests (for Robustness)

conclusions

This form of performance test aims to find performance bottlenecks (such as lock-outs, memory leaks, and thrashing) caused by a small number of Vusers contending for the same resources.

Each run identifies the minimum, average, median, and maximum times for each action. This is done to make sure that data and processing of multiple users are appropriately segregated.

Such tests identify the largest burst (spike) of transactions and requests that the application can handle without failing. Such loads are more like the arrival rate to web servers than constant loads.

1. Identified performance bottlenecks (such as lock-outs, memory leaks, and thrashing) caused by a small number of Vusers contending for the same resources.
2. Ensured that data and processing of multiple users are appropriately segregated.
3. Identified the largest burst (spike) of transactions and requests that the application can handle without failing. Such loads are more like the arrival rate to web servers than constant loads.

C. Volume Tests (for Extendability)

conclusions

This form of performance testing makes sure that the system can handle the maximum size of data values expected.

These test runs measure the pattern of response time as more data is added.

These tests make sure there is enough disk space and provisions for handling that much data, such as backup and restore.

1. Quantified the degradation in response time and resource consumption at various levels of simultaneous users. This is done by gradually ramping-up the number of Vusers until the system "chokes" at a breakpoint (when the number of connections flatten out, response time degrades or times out, and errors appear).
2. Determined how well the number of users anticipated can be supported by the hardware budgeted for the application.
3. Quantified the "Job flow balance" achieved when application servers can complete transactions at the same rate new requests arrive.
4. Ensured that there is enough transient memory space and memory management techniques.
5. Make sure that admission control techniques limiting incoming work perform as intended. This may include extent of response to Denial of Service (DoA) attacks.

D. Stress / Overload
Tests (for Sustainability)

conclusions

This form of performance testing determines how well the number of users anticipated can be supported by the hardware budgeted for the application.

This is done by gradually ramping-up the number of Vusers until the system "chokes" at a breakpoint (when the number of connections flatten out, response time degrades or times out, and errors appear).

During tests, the resources used by each server are measured to make sure there is enough transient memory space and adequate memory management techniques.

This effort makes sure that admission control techniques limiting incoming work perform as intended. This includes detection of and response to Denial of Service (DoA) attacks.

1. Quantified the degradation in response time and resource consumption at various levels of simultaneous users.

2. Determined how well the number of users anticipated can be supported by the hardware budgeted for the application.

3. Quantified the "Job flow balance" achieved when application servers can complete transactions at the same rate new requests arrive.

4. Ensured that there is enough transient memory space and memory management techniques.

5. Make sure that admission control techniques limiting incoming work perform as intended. This may include extent of response to Denial of Service (DoA) attacks.

E. Fail-Over
Tests (for Resilience & Recoverability)

conclusions

This form of performance testing determines how well (how quickly) the application recovers from overload conditions.

For example, this form of performance testing ensures that when one computer of a cluster fails or is taken offline, other machines in the cluster are able to quickly and reliably take over the work being performed by the downed machine.

This means this form of performance testing requires multiple identical servers to be configured and using Virtual IP addresses accessed through a load balancer device.

1. Determined whether the application can recover after overload failure.

2. Measured the time the application needs to recover after overload failure.

F. Spike
"Peak-Rest" or
"Daily" Tests

conclusions

This form of performance testing involves suddenly adding the maximum sustainable load and then returning to a lower level of load to determine whether the app can obtain memory quickly, and then release that memory when no longer needed.

Such runs can involve a "rendevous point" where all users line up to make a specific request at a single moment in time.

Such runs enable the analysis of "wave" effects through all aspects of the system.

Most importantly, these runs expose the efficacy of load balancing.

article article

1. Determined — through suddenly adding and then completing transactions — that the app releases memory.

F. Endurance
"Soak"
"Longevity"
Tests (for Reliability)

conclusions

This form of performance testing makes sure that the system can sustain -- over at least a 24 hour period -- a consistent number of concurrent Vusers executing transactions using near peak capacity.

Because longer tests usually involve use of more disk space, these test runs also measure the pattern of build-up in "cruft" (obsolete logs, intermediate data structures, and statistical data that need to be periodically pruned).

Longer runs allow for the detection and measurement of the impact of occasional events (such as Java Full GC and log truncations) and anomalies that occur infrequently.

These tests verifies provisions for managing space, such as log truncation "cron" jobs that normally sleeps, but awake at predetermined intervals (such as in the middle of the night).

1. Ensured that the system can sustain over at least a 24 hour period a consistent number of concurrent Vusers executing transactions using near peak capacity.

2. Measured the pattern of build-up in cruft (logs, data structures, and statistics that need to be periodically pruned).

3. Detected the impact of occasional events (such as automatic cache flushes, Java Full GC and log truncations) and anomalies that occur infrequently.

4. Make sure there is enough disk space and provisions for managing space, such as log truncation jobs that only occur automatically in the middle of the night.

H. Scalability
(Efficiency) or
Reconfiguration
Tests

conclusions

This form of performance testing involves repeating tests above on different server/network hardware configurations to determine the most cost-effective option to support targeted load levels (one aspect of Capacity Planning).

The outcome of scalability efforts feeds a spreadsheet to calculate how many servers the application will need based on assumptions about demand.

1. Determined — through repeated tests on different server/network hardware configurations — the most cost-effective option to support targeted load levels (one aspect of Capacity Planning).

I. Availability
(Schedulability)

conclusions

This form of performance testing provides a continuous assessment of the availability and speed of key user actions.

These are run on applications in production mode.

This provides alerts when thresholds are reached and trends to guage the average and variability of response times.

The Organizational Context of Load Testing: Capacity Management

Load Testing is relatively new profession. So more often than not, there is misunderstanding.

Testing is important because it is:

• a means to obtain objective quality metrics about systems in their target environment.
• the central means to relate requirements and specification to the real system.

But testing is

• Rarely practiced
• Unsystematic
• Error-prone
• Considered being destructive
• Uncool ("If you are a bad programmer, you might be a tester")

Some (mistakenly) think that there is no need for capacity planning and performance engineering since hardware has become so cheap that it takes more time and effort than to "simply" over-engineer architectures or "just" add more when necessary.

However, most of the bottlenecks today are in the software, which are expensive to diagnose. Additionally, many default configuration options are not appropriate for running large servers.

The newness of the profession make its "fit" within organizations subject to individual fears with personal power and control politics overshadowing organizational needs.

In some way, its role is much like people who use wind-tunnels within an aircraft manufacturer. It can be a frustrating politically — if response time is good, you get no credit. If performance degrades, you aren't doing your "job".

I believe that because of the cross-functional coordination nature of capacity management, it best belongs within a PMO (Project Management Office) guiding cross-functional initiatives requiring the cooperation of the entire enterprise, such as ERP, Y2K, and Digitization (elimination of paperwork).

Capacity Management

Capacity is a capability that is required for delivering an agreed-upon performance at the required service level and cost.

A META Group study published in September 2003 reveals that capacity planning is the top critical issue for large enterprises (those with more than 1,000 people), with 33.8 percent of respondents identifying this as a critical issue. This high priority will continue through 2005/2006, escalating consolidation and efficiency demands.

Load testing ensures that demand for computing power can be met by the supply of computing power.

Proactive capacity management balances business requirements with IT resources, so you can consistently deliver quality service at minimum cost while minimizing the risks of higher utilization rates.

Both ITIL and MOF (Microsoft Operations Framework) recognize that CM consists of three sub-processes:

1. Business Capacity Management (BCM) is implemented by
   Demand Management activities responsible for ensuring that the future business requirements for IT services are considered, planned, and implemented in a timely manner. The capacity management staff can achieve this by analyzing current resource utilization of the various IT solutions and generating trends and forecasts. These future requirements come from account management, which constantly probes current and future customer needs.

2. Service Capacity Management (SCM) is implemented by
   Workload Management activities responsible for translating customer demands into workloads required by IT solutions (the various applications used to create the actual solution) so that the required resources can be determined from this analysis. The process translates both current and future demands to workloads.

3. Resource Capacity Management (RCM) is implemented by
   Performance Management activities

Articles on performance of IBM zSeries S/390 mainframes at zJournal discuss use of reports from IBM's RMF (Resource Management Facility) being massaged by IBM's SSR (Spreadsheet Reporter) and data reduction applicatios such as SAS, MXG, and MICS. Also SMF and CMF reports.

Six Sigma

Traditional "Six Sigma" projects aim to improve existing products and processes using a methodology with an acryonym of DMAIC (Commonly pronounced duh-may-ick, for Define, Measure, Analyze, Improve, and Control).

This is one of many Design for Six Sigma (DFSS) methodologies.

The words Identify, Design, Optimize, and Verify in [brakets] are the basis for the acronym to the IDOV methodology for designing new products and services to meet six sigma standards.

I also appreciate the "Define, Measure, Explore, Develop and Implement" steps from the PricewaterhouseCoopers methodology because they treat performance project artifacts with the same controls as "real" developers.

ITIL Service Delivery

Load Testing is a sub-process of the Capacity Management function within the Service Management standards ITIL (Information Technology Infrastructure Library) and its derivatives the BS 15000 and the MOF (Microsoft Operations Framework)

In this pseudo usecase diagram of Service Delivery

1. Business, Customers, and Users
   1. make Queries/Enquiries
   2. and exchange Communications, Updates, and Reports with the
2. Service Level Management function which
   3. defines Requirements from a Service Catalog quantified
   4. into Targets defined in external SLAs (Service Level Agreements) and internal OLAs (Operating Level Agreements) translated into Operating Level Requirements (OLRs).
   5. tracked for achievements by SLRs (Service Level Reports)

3. Availability Management creates an Availability Plan based on Design Criteria. Success at realizing them is tracked in availability Targets/Thresholds reports.
4. Capacity Management creates a Capacity Plan and Schedules from a CDB. Success at realizing them is tracked in reports of capacity Targets/Threasholds reports
5. Financial Management for IT Services creates a Financial Plan based on the Costs & Charges of each type and model summarized into Budgets and Forecasts reports
6. IT Service Continuity Management creates an IT Continuity Plan based on Risk Analysis

7. All these organizational functions create
   • Alerts and Exception Changes
   • supported by Management Tools and Infrastructure
   managed by the Service Reporting function and supported/monitored by
8. Information Security Management

Capacity Plan

The capacity plan is the consolidated output (deliverable) from the capacity management process.

The capacity plan recommends the resource levels and changes necessary to accomplish operating level requirements that support the service level agreement (SLA). The capacity plan includes the cost and benefit of those resources, reports of their compliance to the IT SLA, and the priority and impact of systems and resources on the overall business and the IT infrastructure.

The Capacity Plan documents:

• Actual Capacity Utilization (current levels and service performance of resources being used)
• Desired Capacity Utilization (a forecast future resource requirements based on business requirements and the IT services needed to support them)
• Basis for Budget Planning

Other Approaches

Mark McWhinney's SEI Load Test Planning Process associates the 6 areas of a Load Test Plan in the sequence they are addressed.

Mark McWhinney's Critical Success Factors for Load Test Projects

Mercury's Capacity Planning product webpage (OEM'ed from HyPerformix) by Sivan Metzger, Product Manager

Capacity Management

Organizational Concerns

The presenting problem with scalability issues is different depending on your position in a large organization.
• To developers of software, it could be mean re-architecting which program modules or services performs some aspect of the application's work. Developers tend to start by a profiling report ranking how often specific modules (classes) are executed.

• To the DBA (Data Base Administration) organization, improvement could mean optimization of the proc, such as adding indexes.
  DBAs tend to start by ranking how often specific database procedures hit the production database.

• To data center Operations personnel, it could mean adding hard drives, additional servers, fiber, NAS filer devices, etc. to prevent and recover from crashes.
  Such people make use of logs or summaries of logs produced by servers under their control.

• To quality assurance management which is separate from the above organizations, a scalability improvement project may be a way for executives to assess (audit) the performance of individuals in the other departments.

The performance engineer's role is often misdefined and misunderstood. He or she can be blamed for not providing information, or scorned for providing unfavorable information.

To operations personnel who are used to doing what they want, their support of capacity management efforts can be perceived as additional unneeded intrusion

Naturally, the role of capacity management is reconciling the needs of all parts of the organization mentioned above at the same time at the most overall cost effective" way.

Organizational design issues can setup performance engineering projects and personnel for either success or failure.

Network Performance Management systems/platforms (such as Avesta's Trinity, Loran Kinnetics, Manage.com's Frontline, and NextPoint's S3) have these capabilities:

• SNMP device management software agents gather information from SNMP agents in network devices and systems
• RMON/RMON II or probe links for traffic monitoring agents track the overall performance of network connections
• Response time measurement agents gauge how well applications, databases, and transactions are performing over the intranet.
• Real-time event filtering agents generate warnings and alerts when devices break or traffic conditions deteriorate
• Historical trend analysis agents store performance data over time to generate periodic graphical representations of network health and status.

As organizations move toward virtualization of server capacity, the job of capacity management would naturally be more about monitoring and managing costs. balancing the production network and benchmarking

Operating Level Agreements (OLA)

An operating level agreement (OLA) formalizes agreements between two or more (usually internal) IT entities into measurable service metric "targets" of specific levels of quality and quantity (cost) of prescribed services.

It is similar to but is normally not as formal as SLAs (Serice Level Agreements) with customers. The OLA should have its metrics stored in the CDB.

The META Group anticipates that through 2005, more than half of IT organizations will invest in formalized IT business plans governed by service-level agreements (SLAs). Unfortunately, fewer than 10 percent of IT organizations have a well-defined service-level management process in place today that can accurately and consistently communicate relevant service levels to the business units.

Roles in a Computer Systems Performance Project

Person/Role	Responsibilities
A. Project Financial Sponsor	Provides the source of funds/budget. Identifies the financial parameters of the resources (budget) available to applications and projects. Provides approvals to expend funds and acquire resources (hiring, consultant contracts, etc.). Audits project completion.
B. Functional Expert (Marketing)	Identifies the objectives (requirements) of applications in the corporate portfolio, such as targeted customer response times, etc. Provides expertise on the process employed by real users -- the business requirements of actual users. Predicts the probable usage patterns within the application.
C. Project Manager	Defines the corporate strategies for managing performance and capacity. Recruits and assembles team members, equipment, and other resources. Drafts and publishes plans and reports. Reports to upper management on progress against plan.
D.1. Developement Liasion	Introduces Performance Engineer to members of the development teams in development. Keeps others apprised of release schedules and changes to the personnel and status of projects. Provides step-by-step usage scenarios and valid/invalid sets of data values.
D.2. AUI Developer	Answers questions such as "What triggers this error message?" and "if we submit this set of values in a submit at this point, what is the expected result?"
D.3. Performance Basis Experts	Analyzes and tunes the components and configurations web server (IIS, etc.).
E.1. Database Performance Basis Expert	Analyzes and tunes the database server for SQL calls from the web server.
E.2. Database Administrator	Provides "show plan" and usage statististics before and after changes. Load and Reload data into test databases used during test runs. Manages backup for assets created during performance engineering.
F. Functional Testers	Provide a spreadsheet listing each screen of the application. Identify what aspects of the application does not work. Identify need for volume testing. Alert if transaction times are noticibly less or more than anticipated.
G. Performance Engineer	Evaluates application risks and determines measurement and test approach, in accordance to pre-defined strategies and project plans. Creates multi-user load scripts, conducts runs, generates analysis (provides expertise on LoadRunner) Identifies areas of concern. Provides technical status on testing activities. Provides developers expertise on the use of performance-impacting techniques.
H. Production Operations (Service Management)	Designs and installs monitoring in the production environment. Publishes production usage and load statistics. Issues alerts when monitors identify alert trigger points.

Capacity Manager Role

The MOF Team Model defines the capacity manager role as part of the Infrastructure Role Cluster responsible for conducting long-term planning to ensure that plans are in place to meet the new and changing requirements of running the business from a networking, telecommunications, hardware, and software perspective.

The capacity manager role oversees the allocation and delivery of service capacity to users. The capacity manager is responsible for planning, monitoring, and reporting activities relating to system and solution capacity, performance measurement, and forecast in the IT organization.

Activities associated with the capacity manager often include:

• Forecasting future service capacity requirements.

• Ensuring that capacity targets can be achieved at a reasonable cost.

• Assisting in the creation and review of new service level agreements (SLAs).

• Providing consulting expertise for the review and creation of any external contracts that include capacity clauses.

• Use of modeling tools and other capacity planning techniques.

The capacity manager is also responsible for managing the day-to-day capacity requirements of services, including:

• Establishing batch job standards (batch skeleton).

• Establishing the batch schedule.

• Changing the scheduling calendar (for example, no runs during holidays).

• Making changes to the batch schedule and testing the new schedule.

• Monitoring the batch process.

• Monitoring the success or failure of batch jobs.

• Monitoring resource availability and performance.

• Capturing metrics and creating performance reports.

• Tuning batch-processing performance and submitting requests for change as necessary.

• Troubleshooting and correcting or escalating job errors.

• Assisting incident/problem management in the correction of errors that are escalated.

• Reviewing, testing, and conducting as-needed requests.

• Submitting RFCs for schedule changes and the addition or removal of jobs from batch runs.

• Ensuring high availability (24x7).

A sample list of Essential Functions (Responsibilities) in a Capacity Manager job description:

SLA-Capacity-Performance Management has overall responsibility for ensuring that there is adequate IT Capacity & Performance to meet required levels of service and for ensuring that senior IT management is correctly advised on how to match Capacity-Performance and demand with SLAs, and to ensure that use of existing Capacity-Performance is optimized with SLA priorities. SLA-Capacity-Performance Manager is also responsible for driving the Service Level Management process for appropriate service levels or service level options with global architecture, line of business application owners & relationship managers, and enterprise software solution platform leaders.
• Implement and maintain the SLM process to the level required by the parent organization.

• Define and deploy the SLA/Capacity/Performance Management process within the organization and be accountable for execution and compliance of the process across the IT support organization

• Ensure IT Services are designed to deliver the required levels of SLA/Capacity/Performance required by the business

• Provide a range of IT SLA/ Capacity/Performance reporting to ensure that agreed service levels of performance, reliability and maintainability are measured and monitored on an ongoing basis

• Optimize the SLA/ Capacity/Performance of the IT application Infrastructure to deliver cost effective improvements that deliver tangible benefits to the business and User including reduction in the frequency and duration of Incidents

• Ensure shortfalls in IT SLA/Capacity/Performance are recognized and appropriate corrective actions are identified and progressed

• Create and maintain a forward looking SLA/Capacity/Performance Plan aimed at improving the overall performance of IT Services and application Infrastructure components to ensure that existing and future business SLA requirements can be met.

• Ensures that appropriate levels of measurements of resources and system performance are set, and that the information recorded in a Capacity Data Base is kept up-to-date and used by all parts of the SLA/Capacity/Performance Management process

• Produces SLA/Capacity/Performance Reports & Plans in line with the organization's business planning cycle, identifying SLA/Capacity-Performance requirements early enough to take account of procurement lead times and/or obtain funding and build out contingency capacity

• Performs process to size all proposed new systems to determine the computer and network resources required, to determine hardware utilization, performance service levels and cost implications

Artifacts and Information Flows among Roles

This pseudo usecase diagram summarizes the information (artifacts) flowing among people assuming certain roles involved in managing the performance of large applications:

Before any Project:

1. Management (C.) defines Roles to resources assigned to the project in accordance with Corporate Performance and Capacity Management Strategies (an extension of this document adapted to an organization) sets the process and limits for all performance engineering activities and projects (such as the rationale for tools acquired, equipment scheduling processes, etc.)

   During planning:

2. Estimated Market (User) Usage Patterns and
3. Budgeted Costs of provisions
   • for the project and
   • for The application/system under test
   both reconciled in a:
4. Performance Engineering Plan for each application/project (a part of each application's development plan), which includes Project Workflow Steps appropriate to Project objectives customer requirements, characteristics of the product under test, and the interactions among them. The technological aspects of this plan is drawn from
5. Performance Tuning Ideas and Techniques used both by developers and by the performance engineer to create

   During construction:

6. Data Configurations and Application Releases are installed and exercised using
7. Test Scripts & Run Parameters which yields
8. Performance Test Results and Reports which are the basis of
9. Simulated Load Patterns captured in a
10. Capacity Model (built using MS-Excel spreadsheet or HyPerformix software) to identify
11. Predicted Performance Profiles for anticipated loads and
12. Alert Trigger thresholds to planned Upgrade Points

    After deployment:

13. Actual Load Patterns can be identified in response to
14. Actual Usage Patterns, which may result in
15. Off-estimate alerts

Project Mission and Objectives for Customer Satisfaction

WHATs - Requirements Critical to Satisfaction	Average Weight (1-5)	Casual User	Exp. User	Sys Admin
1.1 fast to load	4.5	4	4	5
1.2 quick response after submit	4.3	4	3	5
2.1 accepts batched transactions	3.3	0	2	5
2.1 dependable	3.1	4	4	5
3.1 Does not timeout	1.9	1	3	2
4.1 Quick to Recover	1.9	1	3	2

Organizations implementing Design for Six Sigma (DFSS) are wise to scope projects in terms of Customer Requirements (also called "demanded" wants/delights satisfying needs). Kai Yang[2] calls these Critical-to-satisfaction (CTS) items.

The column to the right of each requirement contains weight ratings that allow certain customer requirements to be weighted higher in priority than others in the list. The example shown here is the average of weights for different sub-groups. The "(1-5)" range in this example can be optionally replaced with ISO/IEC 14589-1 evaluation scales or advanced methods such as Thomas Saaty's "Analytic Hierarchy Process" used to establish scales with precise scales:

Models, Methods, Concepts & Applications of the Analytic Hierarchy Process, with Luis G. Vargas (Springer; November, 2000)
Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World (3rd ed. May 1, 1999)

The customer sub-groups shown in this example is for roles working with a computer application:

• Casual users (who usually prefer sessions to timeout automatically and don't care about batch transactions)
• Experienced / Power users (who need to batch several transactions with each submit)
• System/Data Administrators
Alternately, each item may be rated against competitors and alternatives in a "benchmarking" study, from which gaps (marketplace strengths and weakenesses) can be identified and quantified.

QFD graphic programs can add:

• a vertical line graph to illustrate differences in ratings among sub-groups or competitors.

• a triangle to the right of these items to flag relationships among requirements: Related requirements are noted with "+". Conflicting (mutually exclusive) requirements are noted with "-".

HOWs - Product Requirements - Technical Engineering Design Characteristics

At the heart of the Quality Function Deployment (QFD) approach is a matrix of how well each customer requirement is satisfied by measurable product requirements (also called by various authors product design engineering characteristics) that are:

The International TechneGroup, Inc. (ITI) approach for Concurrent Product/Manufacturing Process Development breaks this "WHATs" of the "voice of the customer" (VOC) down further into User Wants, Must Haves, Business Wants, and Provider Wants.

• critical-to-quality (CTQ) to users, such as:
  • Average response time to fully load a web landing page from a root URL request.
  • Average response time to accept/update a user registration page (of 10 data fields)
  • Average response time to complete a login.
  • Average response time to obtain a 20 item result page from a typical search (of 3 key values)
  • Average user time to complete a booking (or other specific business process)
• critical-to-delivery (CTD), such as:
  • Number of seconds from failure (through reboot and startup) to acceptance of first transaction.
• critical-to-cost (CTC), such as:
  • "Cumulative number of transactions before manual intervention is required (for data purging, etc.)"
  • "Number of peak transactions per hour per application server"

The CMM (Capability Maturity Model developed at Carnegie Mellon University) has 7 measures:

1. Need Satisfaction measures (Effectiveness, Responsiveness, Correctness, Versatility)
2. Performance measures (Dependability, Efficiency, Usability, Fidelity)
3. Maintenance measures (Maintainability, Understandability)
4. Adaptive measures (Interoperability, Portability, Scalability, Revisablility)
5. Organizational measures (Cost of ownership, Productivity)

Software Quality Requirements

Associated with SEI/CMU's Taxonomy of Quality Measures, the 2000 revision to ISO/IEC FDIS 9126:1991 and SQuaRE defines 3 types of software quality requirements:
• quality-in-use: the user's view of the quality of the software product when it is used in a specific environment and specific Context-of-use — the extent users can achieve their goals in a particular environment. Its characteristics:
  • Effectiveness
  • Productivity
  • Safety
  • Customer Satisfaction

• External quality: of the product, and
• Internal quality: applicable to interim products such as documents, and source code.

Quality Characteristic	Sub-characteristics	Definition: Attributes of software that bear on the ...
Functionality	Suitability	presence and appropriateness of a set of functions for specified tasks.
	Accurateness	provision of right or agreed results or effects.
	Interoperability	Attributes of software that bear on its ability to interact with specified systems.
	Compliance	Attributes of software that make the software adhere to application related standards or conventions or regulations in laws and similar prescriptions.
	Security	Attributes of software that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs or data.
Reliability	Maturity	frequency of failure by faults in the software.
	Fault tolerance	Attributes of software that bear on its ability to maintain a specified level of performance in case of software faults or of infringement of its specified interface.
	Recoverability	capability to re-establish its level of performance and recover the data directly affected in case of a failure and on the time and effort needed for it.
Usability	Understandability	users' effort for recognizing the logical concept and its applicability.
	Learnability	users'effort for learning its application.
	Operability	users'effort for operation and operation control.
Efficiency	Time behaviour	Attributes of software that bear on response and processing times and on throughput rates in performances its function.
	Resource behavior	amount of resource used and the duration of such use in performing its function.
Maintainability	Analyzability	effort needed for diagnosis of deficiencies or causes of failures, or for identification of parts to be modified.
	Changeability	effort needed for modification, fault removal or for environmental change.
	Stability	risk of unexpected effect of modifications.
	Testability	effort needed for validating the modified software.
Portability	Adaptability	opportunity for its adaptation to different specified environments without applying other actions or means than those provided for this purpose for the software considered.
	Installability	effort needed to install the software in a specified environment.
	Conformance	Attributes of software that make the software adhere to standards or conventions relating to portability.
	Replaceability	Attributes of software that bear on opportunity and effort using it in the place of specified other software in the environment of that software.

ISO/IEC 14598 gives methods for measurements, assessment and evaluation of software product quality.

SPICE - Software Process Improvement and Capability dEtermination is a major international standard for Software Process Assessment. There is a thriving SPICE user group known as SUGar. The SPICE initiative is supported by both the Software Engineering Institute and the European Software Institute. The SPICE standard is currently in its field trial stage.

Project Numerical Business Goals

Capacity planning saves money by balancing two conflicting conditions:

• the expense of unused capacity (in specific componets or system-wide) and, on the other extreme,
• the loss of profits from not having enough capacity to meet demand, especially during busy periods such as during Christmas shopping, Superbowl, and other peak seasons and events.

"Balanced Scorecard" Metrics

The "Balanced Scorecard" (BSC) was introduced in 1996 by a popular book written by Robert Kaplan and David Norton (consultants and professors at the Harvard Business School).

This lists the perspectives of a Balanced Scorecard, and some activity and Key Process Indicators (KPIs) which are most relevant to capacity and performance management:

Perspective & Core Measures	Sample Metrics	Relevant Metrics
Customer How do our customers see us? Market share Customer acquisition Customer retention Customer profitability Customer satisfaction	Satisfaction, retention, market, and account share Over 12 months old customer sales Under 12 months old customer sales Reports on contact with larger customers Sales of new/improved products	# Customers (Users) # Customer Interactions (linked transactions) % User Wait Times vs. an expected statistical profile Response Time for various user actions Network delay Time from various locations User Think Time between specific user actions Abandonment rate per interaction % Returning Customers conducting interactions % Complaints per million transactions
Financial (Results) How do we look to shareholders? Return-on-investment/ economic value-added Profitability Revenue growth/mix Cost reduction productivity	Return on investment and economic value-added: Sales Gross margin after external purchases Gross wages as % of sales End month sales ledger End month purchase ledger Minimum cash available during month	% Asset Utilization: % of capacity utilized during time frame # Hours worked Revenue, Cost, and Profit Per Customer Revenue, Cost, and Profit Per Customer Interaction
Internal (Efficiency) What must we excel at?	Quality, response time, cost, and new product introductions: Sales per Employee "Value added" per Employee % days lost to illness	% reserve capacity available # Planned labor hours spent (on maintenance) # Unplanned labor hours spent (fixing breakdowns) Predictability: % Unplanned vs. Planned labor hours Manageability: # Unplanned events (such as server reboots & restores) Reliability: Mean Time Between Failures or other Unplanned Events Maintainability: Mean Time To Repair or other action (such as upgrade a server) Controllability: % of unplanned events expected statistically
Learning and Growth (Agility, Innovation) How can we continue to improve and create value? Employee satisfaction Employee retention Employee productivity	Employee satisfaction and information system availability: No. of current improvement projects No. of benchmarking visits No. of new/improved products introduced % of Employee days on training/learning	# innovation hours planned % of total time spent on innovation (20% at Google) % of capabilties (metrics) available as planned.

These Balanced Scorecard metrics imply these business strategies:

• To increase product quality and reliability, invest time on innovation rather than mere maintenance.
• Find a way to maintain a precarious balance between seemingly conflicting ends:
  • To maintain customer satisfaction:
    keep response times within a statistically expected profile by keeping a percentage of reserve capacity available (for surges in business growth) so that the abandonment rate and rate of complaints remains low.
    Yet
  • To decrease project cost (labor hours):
    achieve a higher percentage of asset utilization.
• To increase profit, increase the percentage of returning customers.
• To shorten project cycle time, reduce the number of unplanned events.
• To decrease project risk, increase the percentage of hours that are planned vs. unplanned.

Management Dashboards

To each metric, dice and slice:

This ... Last ... This vs. Last ... Cumulative ...

Hour Day Week 4 Weeks Month Quarter Year 2 Year Decade

by Customer by Product/Service by Organizational Level by Location by System/Application by Asset by Transaction Type

Performance Project Plan

Download this MS Project 2003 file containing a sample performance engineering project plan.

This information supplements (and in some cases contradict) the body of knowledge for QAI Certified Software Project Managers (CSPMs).

Performance Within Development LifeCycles

Parasoft's Automated Error Prevention (AEP) process includes several test-friendly steps:
• Unit Test between Code and Integrate
• Integration Test after Code
• Load Test before Deploy
• Load Test after Deploy
• Analyze Performance

Test Type	Timing (When)
A. Speed	Parallel with coding construction, as this provides developers feedback on the impact of their choice of application architecture.
B. Contention
C. Data Volume	On each release when app components are being integrated.
D. Stress/Overload	Pre-Production for each new application version or hardware configuration.
E. Fail-over	Pre-Production for each new application version or hardware configuration.
G. Scalability	Pre-Production for each new application version or hardware configuration.
H. Availability	In Production for each new application version or hardware configuration.

The Context of Performance and Scope of Performance Tuning

Performance improvement and management projects can be considered in the context of these architectural layers and components:

	Context	Components	Tuning Options
A.	Business	Products Departments and hierarchy Individual Users & Permissions	Number and types of users Business hours Batch (cron) jobs Report schedules
B.	Applications	Systems (HR, Accounting, Marketing, Sales, Legal, etc.) Databases (Products, Locations, Customers, Vendors, Employees, etc.) Front-facing Server apps (StoreFront, Login, etc.) Back-end Management apps Client apps (installer, DLLs, JVM)	Software architecture Database layout (schemas) Architecture: N-tiered legacy application-oriented architecture (AOA) or service-oriented architecture (SOA) that may includes a virtualized deployment. Access and security methods
C.	Operating System	Paging File Parameters	File size & location Kernel tuning OS revisions Disk volume layouts
D.	Server Hardware Devices	See Provisions below	number of CPU's vs. App threads Disk (local, NAS/NSF filers) RAM (availability and usage)
E.	Telecommunications Infrastructure	InterNetwork backbone (weather) WAN Routers External Firewalls Switches Telephone, faxes, and other communications equipment Handheld wireless devices	Telecom Network architecture
F.	Data Center Operations	Facilities: Buildings, Furniture, Electrical, Lighting, Plumbing, HVAC, Wiring for Phones, Data, Speakers	NOC (Network Operations Center) Backup/restore strategies Capacity Management Security

CMG (the Computer Measurement Group)

Provisions for Performance Testing

One of the key "Success factors" of a performance measurement project is the availability of resources when needed. Waiting for resources (or working around the lack of resources) is one of the major reasons for project delays.

There are two areas of provision (two sets of budgeted costs):

• Provisions for the application under test (AUT)
• Provisions for the project analyzing the AUT

I. Support Expertise

Time from each role:
1. Executive Sponsor
2. Engagement Manager
3. Performance Engineer
4. Database consultant
5. Development consultant
6. Operations/Network/Security consultant

II. Knowledge

1. Infrastructure Typology describing each server (the versions of software in it, the data it uses and creates, the protocols and frequency of communications between machines)
2. Database schemas and how software access them.
3. Installation and configuration notes for each server.
4. Requirements, design documents, and developer notes on application logic.
5. Instructions to end users (help screens, User Manuals, trainer notes)
6. Sequences of user actions (use cases cross referenced to associated load testing scripts and software component names.
7. Results from Functional Testing ( defect reports on what doesn't work and workarounds)

III. Hardware

There are several dimensions in the IT infrastructure typology and thus cost of providing IT services:
8. Environments (clusters of machines), such as
9. Specific machines used by the application
10. Specific machines used for load testing
11. Component Resources within servers

Environments

• Live Production Environment
• Production Staging / Failover Environment
• Training Environment
• Load Test Environment
• Functional QA Environment
• Development Environment

Specific machines on the technology "stack"

• LDAP and authentication servers.
• Load Balancer Virtual IP addresses (VIPs)
• Ancillary servers and services such as Spam filters, firewalls, and other appliances
• Web servers
• Application servers (such as WebLogic or Websphere servers).
• Incoming/Outgoing Messaging (email) servers
• Web service servers (such as media distribution servers)
• Database servers housing Oracle or SQL servicing database queries
• External storage (NAS/SAN filers) Central file repositories
• Printers, scanners, and other peripheral devices
• Accelerator boards (for encryption, etc.), if applicable

Machines Specfic to the Load Test Environment

• Workstations for tester to develop scripts and scenarios. Tester productivity is greatly improved with multiple monitors.
• Controller collecting dynamic monitoring data.
• Web server to hold load test results and analysis reports (so that viewing of results do not disturb tests being run)
• Load Generator agents (injectors) emulating clients under test
• Loadtest servers emulating servers under test. Examples:
• Loadclient machines to access the test network and to stage application files.

Component resources within each server

Resource	Type	Metric
Motherboard	Volatile	Peak % CPU (CPU Cycles)
Memory	Volatile	Peak KB Cache
JVM Heap	Volatile	Peak KB Usage
Local Hard Disk	Static	Peak KB/sec I/O Bandwidth
Shared Archive Space	Static	Peak GB Peak Usage
DB Disk Space	Static	Peak GB Peak Usage
KB/sec Network	Volatile	Peak Network Bandwidth
OS file handles	Volatile	Peak # file handles
Shared server	Volatile	Peak # J2EE Containers

IV. Data and permissions

14. On the LAN firewall, open port 54345 between the LoadRunner controller and its load generators.
15. Accounts and passwords to test environment machines.
16. Accounts and passwords for administrators, users, and testers of the application under test.
17. Test data extracted (and sanitized) from production.
18. Test data to be generated.
19. Secure certificates, if any.
20. Server images (backups)

V. Software

Specific tools are described and compared in my Load Testing Products
19. Installation package storage and target locations (folders and file names).
20. Network performance (instantaneous response time, throughput, & hop count) checker QCheck freeware download (a subset of IxChariot) — installed as "performance endpoints" on servers being tested/used
21. Access to source libraries (in CVS/SourceSafe).
22. Special programming associated with the application under test:
    • Program(s) to prepare data needed for testing.
    • SQL calls to clear data destroyed during each run.
    • Driver programs for testing calls to lower-level components.
    • Stub programs to emulate availability of lower-level components.
    • Logic that LoadRunner scripts need to emulate internally (such as data siging).
23. Special programming associated with the LoadRunner product:
    • Utilities to archive/backup and restore test assets (in and out of ClearCase).
    • Utilities to archive/backup and restore test results.
    • Utilities to transfer data between LoadRunner and external utilities such as graphing
    • Utilities to display test results and analysis.

Risk Contingency Adjustments

Potential Obstacle / Risk	Likelihood	Avoidance / Mitigation
1. Servers not available early during the project.	High (80%)	a. Use dev. environment to develop single-user scripts.
2. Difficulty with Controller licensing, capacity, etc.	Medium (50%)	b. Identify issues early by beginng to use controller as soon as the first small script (such as login only) is coded.
3. Developers not available	Medium (50%)	c. Perform thorough system analysis to identify issues before scripting. d. develop scripts with likely issues early.
4. Not enough capacity in front-end (portal/login) servers.	Medium (40%)	e. Quanitfy capacity of front-end servers with login_only scripts.
5. Change of personnel during the project.	Medium-High (60%)	f. Take notes. Conduct formal peer walk-throughs. g. Make assignments for skill development.
6. Servers become unavailable late during the project	Low (20%)	h. Use production