HP Quality Center : How To Make It Work This is an unabashed review of how to use HP Quality Center (to version 9.2) despite annoyances. This page is one in a series for QA Managers: Quality Management Maturity Automated Functional Software Testing Test Case Analysis Transition Testing Data Driven Test Frameworks Test Plans Reporting Bugs (Defects) Software Testing Industry

Site Map   About this site   Enter your search terms Submit search form Web   WilsonMar.com

What is Quality Center?

Quality Center (QC for short) is a software system offered by the Software division of HP (Hewlett-Packard), one of the largest corporations in the world.

QC was one of several products (along with LoadRunner, WinRunner, Quick Test Pro, as well as IT governance products) HP purchased as part of its acquisition of Mercury Interactive in 2003.

QC integrates management of Requirements, Test Plans, Test Execution, and Defects into a single system. This ambition makes it popular within corporations familiar with the "waterfall" approach of software developement. This expansive scope is reflected in the Quality Center product being renamed from "Test Director", a name still being used as a subset.

But despite this name change, many workers see the product as being designed to be primarily a device to input data for management reporting (a fancy timesheet) rather than as a tool for collaboration among workers. One of the Agile Manifesto is to prefer "customer collaboration over contract negotiations".

Due to the amount of manual effort necessary to create management reports may make those down on the ground doing the work may feel they must make the "Sophie's Choice" of either disappointing executives who bought Quality Center (to "make your lives easier") or getting the real work (of testing) done.

The Annoyances (and what to do about them)

This article's goal is to not to merely whine but to enable workers in the "trenches" use Quality Center to make their managers happy.

The decision to adopt Quality Center is often made between just C-level executives and HP salespeople who pitch how managers can get better visibility and control of the testing process. This management-oriented "business proposition" and "payback" sounds so enticing that executives often don't feel a need to get full input from technical people or to evaluate "minor technical" details (imagine the salesperson's hand waving). Why inject negativity?

This article warns of potential pitfalls and ways to avoid or work around them:

• Edits and Deletes are Applied Without Saving a Backup Version by Default
• The default workflow assumed by QC results consumes more disk space (and effort) than one would assume.
• Standard reports lack depth of analysis and personalization.
• Making QC a tool for collaboration

Backup On Your Own, But Can't Delete

Quality Center (at version 9.5) requires both a database server and a Windows 2003 Server, not Windows XP nor Vista. So to prevent people messing up the production environment with the result of experiments, a dev environment needs to be set up for QC admistrators to test out patches. This is necessary because patches have been known to fail and cause major problems.

The dev environment is in addition to production and an environment for production staging and fail-over. HP's dashboard requires yet another database.

To learn the system, users need a project with the set of data described in HP's User Guide documentation.

• Make the name of the training (sample) project the same as in the manuals. However, make a copy of it before anyone touches it. This pristine copy can be occassionally copied on top the training project when it becomes trashed.

QC does not come with a Version Control system of its own. However, the QC database does have fields to accomodate versions. But until one is integrated into QC, each change in data wipes out whatever was in the database, with no hope of recovery.

Restore from a complete database backup is cumbersome because it's an "or or nothing" restore. So most sites create a stand-by environment to handle fail-over.

• Given this default restore capability, organizations have gone to preventing "ordinary people" from being able to delete data. They must move items to a hierarchy item such as "zzz_deleted_trash" Then, only people who "really know what they are doing" can permanently delete items moved there. This leads to a whole set of unintended consequences (in addition to the emotional resentment), such as extra overhead on defining and setting permissions on who can do what, plus more complexity (and mistakes) in SQL to generate reports.

Managing Database Growth

As with any database, the Quality Center database must be watched to ensure that it is built with the correct number of extents from the start and that those extents are extended as planned. This is to avoid disruptions from the database crashing when it unexplectedly runs out of disk space.

• When the database is first setup, I recommend that (before doing any real work), do an experiment — setup a very small extent and alerts, then allow the database to run of room. This way you can see whether those who are supposed to receive alerts are indeed getting them and respond to them.

To anticipate turnover in DBA staff, have alerts sent to both the current DBA and the group distribution of system administrators and their managers.

• I have a set of SQL queries (and spreadsheet for graphing) that is run each week to obtain and display the trend of row counts in key tables within the Quality Center database.

All this is necessary becasuse the default workflow (business process) causes more disk space to be consumed than one would assume, due to the linkages among items Quality Center assumes is needed. For example, each test item must be copied to Test Lab.

The Fallacy of Coverge Percentage

Yes, Quality Center comes with a lot of reports and charts. And it does come with a separate Dashboard that can be installed (on an additional server using an additional database).

Default QC reports and graphs are conceived from a rather technical point of view, providing simple counts without personalization to vary for each manager only the fields, sorting, and filtering relevant to him/her. Simply put, the large complex organizations that typically use Quality Center need more complex reports and graphs than QC provides by default.

The "Coverage Progress" chart (at right) is prominent in the software to summarize progress toward "completion" at several moments over time.

At the last reporting period shown (on the far right at end of the ribbons), 100% of Requirements are shown as "Assigned" by the purple ribbon at the top. But those who have taken a system through implementation know that new Requirements inevitably crop up. So managers need QC to include quantification of "unknown unknowns" (an estimate of how much work might be added along the way).

It's difficult to figure out, but this chart shows the "Planned Coverage" (yellow ribbon) at 60%. Those presenting this chart need to be prepared to answer the naive but valid question of "why were only 60% planned coverage"? Many in our culture assume that 100% (or even 110%) is the usual goal. A skilled manager would ask for an analysis of the economic trade-offs between risk and work. There may not be payoff in doing everything planned, as there is less liklihood of finding a defect.

Another common assumption is that whatever is planned (yellow line) should be executed (dark red line). Practically, however, it is often desirable to have plans for more work in case other items are blocked, which frees up time for additional work elsewhere. However, within the default Quality Center structure, it is difficult to differentiate between items commited to be executed versus items held in reserve.

I think it is a bad management tactic to measure test lab execution on a 100% scale because we want testers to define more tests than they plan on running so that if certain items get blocked, they would have already defined additional tests to run immediately rather than just waiting around. Judging testers based on a default 100% would discourage (indeed penalize) testers for being proactively productive.

This whole issue is why one of the Agile Manfesto items is prioritizing "Responding to change over following a plan".

Agile Ideas

CI tools include Selenium, FitNesse, Cucumber, and Concordion.
Here is where a paradigm shift occurs with the Agile methods that have come into vogue early in the 21st century.

Unlike a "heavyweight" test management approach such as Quality Center managing a backlog of verfication work by testers, the Agile approach finds wisdom in automated "Continuous Integration" (CI), where developers incorporate testing as part of their own work. An Agile developer cannot call a creation "done" unless it's also been tested. When a defect is identified, developers are supposed to stop and fix it immediately. Thus, there is no separate backlog of testing to prioritize and manage. The testing backlog is the same as the development backlog.

In the Agile world, testers are embedded among developers. So issues of "testability" are discussed and solved. 100% coverage is assumed with Agile. This includes testing on various browsers and against various internal conditions.

"Specialized Test Management Systems are an Agile Impediment" —Elisabeth Hendrickson, October 6th, 2009

"... test automation created by a siloed QA team working in isolation to reverse-engineer existing software and automate tests against an untestable UI using proprietary tools accessible only to a few select team members is guaranteed to be incredibly expensive both to create and to maintain, and also ridiculously fragile. People still do that?! :-) No wonder "testers" get such a bad rap. Their mission is literally impossible." — Jason Huggins, April 4, 2008

The "test first" concept with Agile is that code to validate whether the code being developed works before that code is written. Thus, code is written to satisfy verifiable goals. This is a result of the Agile Manifesto of preferring "Working Software over comprehensive documentation".

Linkages Among Components

To meet regulartory requirements, HP Quality Center provides a system to document "traceability" by establishing links for each and every item within a set of complex hierarchies. The set of coverages are illustrated here.

To track "Assigned Requirements" coverage, each requirement (product backlog item) must be linked to some Release Cycle. This makes replanning a cumbersome process. More troubling still is Quality Center provides no direct linkage between Requirements and Test Plans (called Test Sets). Requirements are instead linked to Test Set Folders which contain Test Sets.

To be able to calculate executed coverage, each test run must be manually created and designated with the test set it is based upon.

Metrics Managers Want

The most useful metrics are based on workflow tasks (what people do), and thus serve as predictors of where time and budgets might need adjustment.

Examples of more sophisticated metric reports include:

__% of Test Subject Items cover Requirements
__% of Test Subject items contain Test Steps
__% of Test Subject items selected for execution as Test Sets
__% of Test Sets redefined as Test Instances
__% of Test Instances executed
__ Defects per Test Set,
__ Defects per Test Subject item

What Do You Call That, Again?

Different names for objects are used in different parts of the application. So a certain amount of education is needed to clarify the vocabulary among different names used in several places in the application:

The alphabetical list on the left (with objects in singluar form) is from the menu only admininistors use to define custom fields, which does not occur frequently. It's a good idea that administrators avoid using this as the basis for describing content in the database.

The hierarchy on the right is from TOOLS > Document Generator used to create detail reports. This is perhaps the best structure to use as a basis for defining a common vocabulary.

This matches text such as "Subject" at the top of the Test Plan component hierarchy tree. This also matches the "Test Set" at the top of the Test Lab component hierarchy tree.

The names of tables and columns within the database need to be known to those crafting SQL for Excel reports.

Those coming from or transitioning to other systems (such as ScrumWorks or Microsoft TFS) would need to memorize the equivalence of terminology.

For example, ScrumWorks uses the word "Themes" within "Epics" to organize what needs to be achieved by each release.

Ownership by Department/Workflow

QC currently does not provide a mechanism for restricting actions on specific items. QC does not come with a field that defines the owner of each item, designating the role, organizatonal group, or individual who can delete or change or edit the item.

The result of this missing security functionality is that someone allowed to make deletions can delete any item within a project. This is a big hole in functionality considering that organizations that use QC are usually complex. The work-around:

• In each table, define a user field with a name such as "workgroup" or "owner" to designate who has responsibility for its maintenance. This would establish accountabiity for "clean-up" activities. When reports are filtered by this custom field, the report is applicable to each specific group (much better than requiring eveyone to ignore everyone else's data).

Database Schema

The Quality Center database is rather complex, with dozens of tables. So rather convulted SQL coding is often necessary to produce a usable report.

HP QC 9.2 quality_center_db.chm. [After downloading, in Windows Explorer go into the Properties for the file and unclidk the Read-Only property. If you still have problems reading .chm files, see this)]

Hierarchy Restructure

It is desirable for the hierarchy of test sets and test runs to take several forms over time.

Executives may want a hierarchy by version and release:

Version -> Release -> Type of testing -> WorkGroup -> Item

Those who need to analyze items across workgroups for duplication need an "upside down" hierarchy:

Item > WorkGroup > Test Type > Release > Version


However, work is planned by those responsible for an area of responsibility, called a System or Workstream or PlanGroup:

System > SubSystem > Item > Test Type > Release > Version

• So I've developed scripts to restructure hierarchies based on the value of fields. This is customed for each organization's unique vocabulary.
• Before the restructing occurs, an edit script is run to ensure that each such field contains data valid to construct the hierarchy.
• Ironically, in fear of being allowed to crash the database, those using SQL to manipulate hierarchies are often denied the permissions needed to create temporary tables needed to efficently work with hierarchies.
• Also, although updating the schema to add a hierarchy level number column would make SQL queries easier and much more efficient (and not require temp tables), most organizations avoid doing so out of fear, and thus paralize QC administrators as well.

What Are The Numbers Supposed To Be?

The default reports only say how many of each type of data are in the database. But missing is what number is supposed to be there (at various moments of time during the project)?

Tracking the percentage of test plans (or other items) planned versus entered in the system is where SQL and Excel come in (next section), since this is not a default part of Quality Center.

Many don't bother with additional reports not because of the technical challenges, but because of politics. Putting out a projection would "hold their feet to the fire".

For some reason, most test managers are allowed to say that they can't predict how many defects will be produced. Unfortunately, most sales managers don't have such a luxury. Stockholders and the SEC require them to provide "guidance" each quarter.

Whether it's publicly voiced or not, organizations make some prediction about how many defects they expect in their staffing actions. If NO developer time has been allocated to analyze and fix bugs, then expect NO bugs to be found. If later there is a scamble to find time to address bugs found, then some "mismanagement" has occured.

Skill at predicting future events is a learned skill, and require insightful dialog among development and test managers, something that is lacking in many organizations.

This is why one of the Agile Manifesto items is to prefer "individuals and interactions over processes and tools".

Basic Burn-Down Charts

This gap is answered by software supporting Agile approaches which produce a burn-down chart such as this for points of functionality being developed by "Pigs".

In this example (produced by Rally software), each column illustrates the count of "function points" at the end of each day. The blue column is the backlog of how many remain to do.

In organizations that practice Agile, this chart is reviewed during daily scrums facilitated by a Scrum Master.

Each column defines the work for each sprint of fixed duration (1-3 weeks each).

The starting backlog is defined for each sprint are agreed upon based on a prioritized list of functionality maintained by the Product Owner.

Prioritization of features developed within each sprint is based on estimates of business value and effort obtained during Sprint Planning workshops and Product Backlog Refinement workshops.

The diagonal line in the basic burn-down chart is labeled "ideal" because it represents ideal circumstances (not management expectations). The difference between ideal and actual progress reflect blocks the team encounters.

The rate that features are worked off is (the team's capacity of items each sprint) is called its velocity.

Agile practioners prefer the horizontal time frame (including the length of each sprint) to not change to foster learning needed to improve the accuracy of estimating skill.

Changes To Quality Center Usage for Agile

Quality Center was built to track individual tasks and the hours of work associated with each. It can also differentiate between completed (Accepted) items as green columns count versus items still to be processed as blue columns.

However, an important difference with Agile vs. traditional approaches to project management is that generalized points are used instead of absolute man-days so that estimation bias can be more easily adjusted for reality.

Such an approach requires custom fields in Quality Center.

Rally maintains this chart by establishing a two-way exchange of data with Quality Center.

But this should be done with the understanding that there is a fundamental difference between Quality Center and Agile/Scrum in that Quality Center forces estimates, whereas with Agile an "epic" can remain as such without being associated with a numerical budget of time.

Additional processing (by Rally) adjusts estimated completion dates based on Scrum estimating techniques.

True Agile practioners do not differentiate between testers who file bugs and developers who fix them, since both work toward the same definition of "done". Instead of QA saying "it's done when I say it's done" (after developers say they are done), an interdisciplinary team gets done faster together.

Enhanced Burn-Down Charts

ScrumWorks' Enhanced Burndown chart has a more informative presentation which differentiates "headaches" (work to remove defects) from original effort planned. Some may find this too sophisticated / confusing.

The length of each bar — the total effort remaining after each period — is after a push down the original effort plus a push up to work off the headaches/defects remaining. An enhancement to this are different colors to depict new headaches/defects added during the period.

The red diagonal line displays the Best Fit line depicting the average velocity of burndown. Periods of slower progress would have bars floating above the red line. Periods of greater progress would have bars below the red line.

Producing a burn-down chart does not make an organization "agile" nor does it mean management is "in control". Burn-down charts do not address management of progress in all aspects of software development.

Being Agile takes a change in mindset about responding to change through constant communication and adjustment rather than pre-planned expository and adversarial contracts. An organization does not really "get" Agile a "Scrum Master" or supervisor writes up performance reviews on programmers based on burn-down charts or whether detailed specs were developed according to management expectations.

Becoming Agile: ...in an imperfect world (Paperback) (Manning Publications, June 8, 2009) by Greg Smith, Dr. Ahmed Sidky explains Agile from a case-study perspective within a 9 step process for adoption of agile practices. and presents a Agile Measurement Index

Succeeding with Agile by Mike Cohns

Excel From SQL

• If it is done, SQL would be used to extract counts of what has been entered. This would be imported into Excel, which produces a table or chart comparing actual versus planned counts.

  This is a "poor-man's" visualization capability. The Excel graphic reports I describe on another page are based on the structure of Excel chart files below:



The file "Defect Analysis Data R1 v3.xls" is replaced whenever it is generated from SQL within QC.

Each tab in the SQL within QC is produced as a separate tab in the output spreadsheet.

• Test Sets Counts (TSetCounts)
• Test Subjects Counts (TSubjCounts)
• Test Run Counts (TRunCounts)
• Requirements (ReqCounts), not shown

Another spreadsheet (the Base spreadsheet) references the Data spreadsheet.

A third spreadsheet then produces charts by referencing the Base spreadsheet.

The trouble with this approach (using Excel) is that these need to be run on a specific date (e.g., every Friday at noon) so that counts at each particular point in time can be captured when it's available.

A program to generate the count of each at various points in time can be generated, but would require more complex code.

Reports Generation Options

To it's credit HP does provide many ways to create custom reports. But there are so many different technologies to create reports that a manager may have to use several technologies to get a complete picture.

1. default online lists and individual detail screens
2. default online reports
3. custom reports defined withing QC
4. custom SQL extract to SQL defined withing QC
5. Custom Excel charts created from SQL extrcts
6. VBScript run outside QC
7. default HP Dashboard reports
8. Custom HP Dashboard reports
9. A custom Business Intelligence (BI) application referencing a copy of the QC database

During my consulting engagements, I customize from working generic examples refined over several engagements.

Quality Center needs to be considered within the total set of enterprise applications operating within most large organizations:

The Architecture of External Interfaces

To get serious with managing software, I recommend that one treat data from Quality Center like they do with other corporate metrics, and get it integrated with the corporate dashboard (BI system).

Here is a flowchart of how Quality Center interacts with external data.



Integrations needs to be considered with:

• Email alerts
• Project management systems (such as MS Project, PlanView, etc.)
• Point-in-time static reports archived on SharePoint, etc.

VBScript

HP provides a rich VBScript API to retrieve data programmatically from the Quality Center database.

• HP QC 9.2 OTA API Reference.chm
• HP does not provide a debugger for VBScript programs.
• To debug VBScript programs, enable the Microsoft Debugger for Office.

Data Uploads and Downloads (Import/Export)

QC provides batch import capabilities in add-ins to Excel and Word.

Most people use Excel rather than Word (which requires each element to be marked).

The Excel interface is rather error-prone because importing is based on Excel column letters (A, B, C, etc.) rather than column names as in most CSV files. So here is my advice:

• Standardize on a template that contains only the fields needed for your organization. I have some to show you.
• Use separate templates for importing requirements and test plans.
• Create data edit macros and run them before importing.
• Backup existing data before performing the import.

Certification and Training

• HP Exam HP0-M15 replaces Mercury exams QC82SP, QC90SP, HP0-M11, HP1-M11.
• HP Exam Using Quality Center 9.2 #QC92USING-COURSE-01 - Instructor-led (1 day; $650 US)

Related Topics:
Performance Testing 
NT Perfmon / UNIX rstatd Counters 
Mercury LoadRunner 
Mercury Virtual Table Server (VTS) 
Mercury WinRunner 
Rational Robot 
Free Training! 
Tech Support 

Your rating of this page: Low High Your comments on this topic, please: Publish this comment publicly

Your first name: Your family name: Your location (city, country): Your Email address:  Email me updates

Top of Page Thank you!