Bug Reporting

This discusses what to keep in mind as you write bug reports that make debugging easier.

Topics this page:

App Versions

Site Map
About this site

Application Version

requirements

...

Run Conditions

Platform characteristics (CPU type, RAM memory, etc.).
Type of test run (maximum values, load test, time-out, etc.)
Focus of run (negative test, etc.)
Date and time settings

This gives debuggers the information they need to narrow down the cause of the problem.

...

Steps to Reproduce

reproduce

As user ___ from an NT4 Pro client, sign in the ___ server with a blank database.
Click menu item ...
Select From any account.
Enter ...
Select Hungary.
Click Next.
Notice that ...

Start each step with an active verb such as "click OK", "select checking account", "type value 99 in the Transaction Amount field", etc. The last step should reveal the concern stated in the Summary sentence ("Temp delay EOT"). Note whether "Temp delay" screens results in an End of Transaction or End of Session (EOS) which causes another logon.

To avoid needless repetition, produce a Test Specification document that defines the meaning of special words.

If a special program not used by typical users is needed to view results (such as a log viewer), note its use.

...

Analysis

This occurred for all account types.
This did not occur for other countries (Spain, etc.).
This did not occur for other currencies.

...

Priority/Severity Level

impact

prioritizing

A sample guideline for assignment of Priority Levels during the product test phase include:

Show Stopper - an item that prevents further testing of the product or function under test. Examples of this include an installation process which does not load a component; a GPF (General Protection Fault) or other situation which freezes the system (requiring a reboot); or a missing menu option or security permission required to access a function under test.
Within a production test context, this category could also include incorrect financial calculations and even cosmetic items.
High — an item that does not function as expected/designed. Examples of this include: inaccurate calculations; the wrong field being updated; the wrong rule, phrase, or data being retrieved; an update operation that fails to complete; slow system turn-around performance; or a transaction journal record which fails to occur as expected.
Medium — Annoyances which does not conform to standards and conventions. Examples include incorrect hot-key operation; an error condition which is not trapped; or matching visual and text links which lead to different end points.
Low — Cosmetic issues which are not crucial to the operation of the system. Examples of this include misspelled or ungrammatical text; or inappropriate, inconsistent, or incorrect formatting (such as text font, size, alignment, color, etc.).

Management makes the decision whether an application should be shipped in light of the number and types of defects open.

If you practice FMEA (Failure Mode and Effects Analysis), you would prioritize based on the Risk Priority Number (RPN) number calculated by multiplying three numbers:
RPN = SEV x OCC x DET

Severity (SEV) for how significant the impact of the effect is on the customer.
Occurrence (OCC) for how likely the cause of the failure mode is to occur.
Detection (DET) for how likely the current system is able to detect a cause or failure mode if it does occur.
The higher the RPN, the more focus should be given to the particular problem in the root Cause-and-Effect diagram.

Assign To

Most organizations who value fast turnaround of bug fixes prefer this approach.

However, some (usually complex) organizations prefer all bugs for a project to flow through a coordinator or manager who then assigns the individuals to work on each particular bug.

For example, if the development team is divided by people who write reports and people who write GUI code, analyze bugs so that you can specify who should review each specific bug report.

"GUI dialog xxx does not ... "
"REPORT yyy does not ... "
"REQUIREMENT yyy.xx does not define ... "

This avoids the common situation of a bug report from being closed by one person for a single sub-item when additional sub-items going unsolved.

...

Summary

Be specific. Don't use vague phrases such as "problems with .", "is incorrect", or "issue with". State the expected behavior which did not occur - such as "after . pop-up . did not appear." and the behavior which occurred instead.
Use present tense. Say "appears" instead of "will appear".
Don't use unnecessary words such as "apparently". State observed facts.
Don't add multiple exclamation points!!!! (We do want to help.) End sentences with a period.
DON'T USE ALL CAPS (That's the same as shouting.) Format words in upper and lower case (mixed case).

Other examples are different routes throught he application to the same point of failure, similar functions in the same application, different browsers, different software and hardware configurations, different locales, different security contexts, previous versions, etc. when applicable.

...

Status of Defect Items

Testers create defect items with an Evaluation status. The item remains in that state while the item is being analyzed by developers.
A tester may place a defect item into Withdrawn status items which duplicate another item already reported or is the result of tester misunderstanding about how the application should operate.
Management may wish to mark certain defect items as Discarded or Forwarded to a future release if the application is allowed to be shipped as-is with the defect. This usually happens because of time constraints.
Analysts place a defect item into Development status when analysis is complete and changes are being made to the application by developers.
Developers changes the status of a defect item to Testing after unit testing. This signals to testers that application changes can be tested again. Such items are noted on a Hand-off memo from developers to testers.
If changes by developers did not fix the problem reported, that defect item can be Returned to development. Another defect item is created if a fix introduces another problem.
After a tester verifies that a defect has been fixed, the defect item is Closed.

...

Air Force Maintenance Complaints

From Dumblists.com

Here are some actual maintenance complaints submitted by US Air
Force pilots and the replies from the maintenance crews.

Problem: "Left inside main tire almost needs replacement."
Solution: "Almost replaced left inside main tire."

Problem: "Test flight OK, except autoland very rough."
Solution: "Autoland not installed on this aircraft."

Problem #1: "#2 propeller seeping prop fluid."
Solution #1: "#2 propeller seepage normal." Problem #2: #1, #3, #4 propellers lack normal seepage."

Problem: "The autopilot doesn't."
Solution: "IT DOES NOW."

Problem: "Something loose in the cockpit."
Solution: "Something tightened in cockpit."

Problem: "Evidence of hydraulic leak on right main landing gear."
Solution: "Evidence removed."

Problem: "DME volume unbelievably loud."
Solution: "Volume set to more believable level."

Problem: "Dead bugs on windshield."
Solution: "Live bugs on order."

Problem: "Autopilot in altitude hold mode produces a 200 fpm descent."
Solution: "Cannot reproduce problem on ground."

Problem: "IFF inoperative."
Solution: "IFF inoperative in OFF mode."

Problem: "Friction locks cause throttle levers to stick."
Solution: "That's what they're there for."

Problem: "Number three engine missing."
Solution: "Engine found on right wing after brief search."