QA Metrics That Matter — What to Measure & What to Ignore (2026)
Number of test cases is a vanity metric. This guide covers the 8 QA metrics that predict quality outcomes, plus the 6 vanity metrics that waste leadership's attention — with real dashboards.
Last updated 2026-07-20 · 10 min read · By Avinash K
"How many test cases did we execute this sprint?" is the wrong question. The right questions are about escape rate, MTTR, and coverage of business risk. This guide gives you the 8 metrics that actually predict quality and the 6 vanity metrics to stop reporting.
Key takeaways
- The 8 metrics that predict outcomes.
- The 6 vanity metrics to stop reporting.
- A one-page QA dashboard template.
- How to defend metrics to a skeptical CTO.
1. The 8 metrics that matter
| Metric | Why it matters | Target |
|---|---|---|
| Defect Escape Rate | Bugs caught in prod / total bugs | <10% |
| MTTR (defects) | Time from report to prod fix | <7d for S2+ |
| Flaky test rate | Non-deterministic tests / total | <2% |
| CI pipeline duration (p90) | Dev velocity gate | <15 min |
| Automation coverage of critical paths | Regression risk | 100% |
| Change failure rate | DORA — % deploys that fail | <15% |
| Test env uptime | Blocked hours / total hours | >95% |
| Requirement coverage | Reqs with at least 1 test | 100% |
2. The 6 vanity metrics
- Number of test cases (quantity, not value).
- Number of bugs raised (rewards noise).
- Test execution count (rewards busywork).
- Automation script count (same trap).
- Line coverage without mutation score.
- Time-in-status for tickets (games itself).
3. One-page dashboard template
Top row: escape rate, MTTR, change failure rate (trend + current). Middle: pipeline duration + flaky rate. Bottom: environment uptime + critical-path coverage %. Everything on one screen, refreshed weekly. Use Grafana, Looker, or Retool — do not build it in slides.
4. Defending metrics to a CTO
Tie every metric to a business outcome: escape rate → revenue at risk; MTTR → SLA credits; flaky rate → dev velocity tax. If you cannot connect a metric to money or hours, drop it. Reference: DORA four keys. Related: CI/CD for QA, observability.