Testing metrics can be classified as belonging to one or more of the following categories:
· Product metrics, which measure some attribute of the product, such as the extent to which it has been tested or the defect density
· Process metrics, which measure the capability of the testing or development process, such as the percentage of defects detected by testing
· People metrics, which measure the capability of individuals or groups, such as the implementation of test cases within a given schedule
Metrics related to product risks
Metrics related to product risks include:
· Percentage of risks for which some or all tests fail
· Percentage of risk not yet completely tested
· Percentage of risks covered, sorted by risk category
· Percentage of risks identified after the initial quality risk analysis
Metrics related to defects
Metrics related to defects include:
· Cumulative number reported (found) versus cumulative number resolved (fixed)
· Mean time between failure or failure arrival rate
· Breakdown of the number or percentage of defects categorized by the following:
o Particular test items or components
o Root causes
o Source of defect (e.g., requirement specification, new feature,
regression, etc.)
o Test releases
o Phase introduced, detected, and removed
o Priority/severity
o Reports rejected or duplicated
· Trends in the lag time from defect reporting to resolution
· Number of defect fixes that introduced new defects (sometimes called daughter bugs)
Metrics related to tests
Metrics related to tests include:
· Regression and confirmation test status, including trends and totals for regression test and confirmation test failures
· Hours of testing planned per day versus actual hours achieved
· Availability of the test environment (percentage of planned test hours when the test environment is usable by the test team)
Metrics related to test coverage
Metrics related to test coverage include:
· Risk coverage
· Environment/configuration coverage
Metrics to monitor test planning and
control
Metrics to monitor test planning and control activities may include:
· Defect discovery
· Planned versus actual hours to develop testware and execute test cases
Metrics to monitor test analysis
Metrics to monitor test analysis activities may include:
· Number of defects found during test analysis (e.g., by identifying risks or other test conditions using the test basis)
Metrics to monitor test design
Metrics to monitor test design activities may include:
· Number of defects found during test design (e.g., by developing tests against the test basis)
Metrics to monitor test
implementation
Metrics to monitor test implementation activities may include:
· Percentage of test data records loaded
· Percentage of test cases automated
Metrics to monitor test execution
Metrics to monitor test execution activities may include:
· Percentage of test conditions covered by executed (and/or passed) test cases
· Planned versus actual defects reported/resolved
· Planned versus actual coverage achieved
Metrics to monitor test completion
Metrics to monitor test progress and completion activities will include
a mapping to the milestones, entry criteria, and exit criteria (defined and
approved in test planning), which may include the following:
· broken down by whether they passed or failed
· Total defects, often broken down by severity, priority, current state, affected subsystem, or
· other classification
· Number of changes required, accepted, built, and tested
· Planned versus actual cost
· Planned versus actual duration
· Planned versus actual dates for testing milestones
· Planned versus actual dates for test-related project milestones (e.g., code freeze)
· Product (quality) risk status, often broken down by those mitigated versus unmitigated, major
· risk areas, new risks discovered after test analysis, etc.
· Percentage loss of test effort, cost, or time due to blocking events or planned changes
· Confirmation and regression test status
Metrics to monitor test closure
Metrics to monitor test closure activities may include:
· Percentage of test cases checked into re-usable test case repository
· Percentage of test cases automated, or planned versus actual test cases automated
· Percentage of test cases integrated into the regression tests
· Percentage of defect reports resolved/not resolved
· Percentage of test work products identified and archived
In addition, standard project management techniques such as work
breakdown structures are often used to monitor the test process. In Agile
teams, testing is part of a user story’s progress on the burndown chart. Where
Lean management techniques are used, testing progress on a story-by-story basis
is often monitored by having the user story card move through a column on the
Kanban board.
Given a defined set of metrics, measurements may be reported verbally in
narrative form, numerically in tables, or pictorially in graphs. The
measurements may be used for a number of purposes, including:
· Reporting, to communicate test findings to interested project participants and stakeholders
· Control, to change the course of the testing or the project as a whole and to monitor the results of that course correction
