Executive Definition
A flaky test is an automated test that produces different outcomes — pass or fail — on repeated executions against the same code, configuration, and environment. The variability comes from the test itself or its infrastructure, not from the system under test.
Flakiness is uniquely destructive because it corrodes trust. A test that always fails is quickly investigated. A test that fails one time in twenty is retried until green, then forgotten. Over months, the retry habit spreads across the suite and legitimate failures are dismissed as 'probably flaky'. Google's engineering-productivity research documents this dynamic: once flake exceeds roughly 2%, engineers stop reading test failures at all.
The seven canonical root causes cover the vast majority of flaky tests: implicit waits and race conditions, shared test data or order-dependent tests, network calls with real dependencies, time and timezone assertions, animations and CSS transitions, resource exhaustion in CI, and non-deterministic collection ordering (JavaScript's Object.keys, Java's HashMap iteration). Each has a clean fix — but the wrong fix, universally, is adding a retry.
Retries mask the symptom, protect the wrong tests, and encourage more flakiness. A retried test still catches the bug it was designed for, but the fix cost grows every time the retry kicks in because the reproduction becomes harder. Teams that adopt aggressive retry limits end up with a suite that lies less loudly but no less often.
The correct treatment path is the same regardless of root cause: quarantine, observe, reproduce, fix root cause, and re-admit with a stability gate. In 2026 tooling has caught up — Playwright's trace viewer, pytest-flakefinder, and Google's Flake Bot all automate the observe-and-reproduce phases — but the discipline still lives with the engineer. No tool decides that a flaky test is worth keeping.
Architecture & Production Code
The quarantine loop is the shape of any healthy flaky-test policy. It turns a corroding suite into a self-healing one.
┌────────────────────┐
│ Flaky test detected│
│ (pipeline signal) │
└──────────┬──────────┘
│
▼
┌────────────────────┐
│ Tag @quarantine │
│ Move out of gate │
└──────────┬──────────┘
│
▼
┌────────────────────┐
│ Observe: HAR, │
│ video, trace, │
│ CI resource stats │
└──────────┬──────────┘
│
▼
┌────────────────────┐
│ Reproduce locally │
│ retries=0 │
└──────────┬──────────┘
│
▼
┌────────────────────┐
│ Fix root cause │
│ (never a retry) │
└──────────┬──────────┘
│
▼
┌────────────────────┐
│ Stability gate: │
│ 100 runs green │
│ → back in suite │
└────────────────────┘The first move — quarantine — is the one teams delay the longest. A flaky test that keeps blocking merges creates pressure to 'just rerun and merge', which normalises the behaviour. Auto-quarantine after N failures in M runs eliminates the negotiation.
The observe phase is where modern tooling pays off. Playwright's trace viewer, network HAR captures, and CI-runner resource metrics let you diagnose flakes that used to require dozens of manual reruns. Any test worth keeping earns a full artefact bundle on every failure.
The stability gate is the readmission criterion. A fixed test must run 50–100 times cleanly against a busy CI runner before rejoining the blocking suite. Teams that skip the gate discover the 'fix' was a narrower flake, not a fix.
// playwright.config.ts — treat retries as a diagnostic, not a solution
export default defineConfig({
retries: process.env.CI ? 2 : 0, // 0 locally to see real flake
reporter: [["html"], ["blob"]],
use: {
trace: "retain-on-failure", // always keep failure evidence
video: "retain-on-failure",
screenshot: "only-on-failure",
actionTimeout: 5_000, // fail fast, retry deliberately
navigationTimeout: 15_000,
},
});
// A test that used to be flaky because of a race on the API response
import { test, expect } from "@playwright/test";
test("dashboard shows the freshly created order", async ({ page }) => {
// WRONG: await page.waitForTimeout(1000) ← the classic anti-pattern
const orderResponse = page.waitForResponse(
(r) => r.url().endsWith("/api/orders") && r.status() === 201,
);
await page.getByRole("button", { name: "Create order" }).click();
await orderResponse;
// Await the specific outcome, not a timer
await expect(page.getByTestId("order-row-latest")).toBeVisible();
});Flaky vs Intermittent vs Non-Deterministic
| Term | Meaning | Typical cause | Correct response |
|---|---|---|---|
| Flaky | Same code, different results | Test infrastructure or assertion timing | Quarantine, diagnose, fix root cause |
| Intermittent (real bug) | Real product bug that surfaces occasionally | Race condition in the system under test | File a bug; keep the test as a canary |
| Non-deterministic | Test relies on ordering that is not guaranteed | HashMap iteration, floating point, time.now() | Freeze the source of nondeterminism (fake clock, sorted keys) |
| Environment-flaky | Fails only in CI, passes locally | Resource exhaustion, network egress, container drift | Match local and CI runner specs; add resource limits |
'Flaky' is often used as a catch-all excuse. Separating the four categories forces a specific diagnosis and prevents the retry-and-forget default. Only true test-side flakiness deserves quarantine; real product races deserve bug tickets.
Production Debugging Scenarios
These are the three flake patterns that consume the most engineering hours in typical CI pipelines. Each has a specific fix that outperforms retries by orders of magnitude.
Test flakes only under CI parallelism
- Symptom
- Suite is 100% green locally and on --workers=1; fails ~5% at --workers=4.
- Root cause
- Tests share a database record identified by a fixed ID.
- Fix
- Generate unique IDs per test worker with worker_index, or scope every test to its own transaction that rolls back on teardown.
Date-boundary test fails only around midnight UTC
- Symptom
- Nightly build fails once a week with 'expected 2026-01-15, got 2026-01-16'.
- Root cause
- Test uses new Date() during setup and asserts against a static string.
- Fix
- Inject a fake clock (vi.useFakeTimers with a fixed date) and remove Date.now() from all assertions.
Playwright test flakes on animated modal close
- Symptom
- Modal-close button reports 'element not stable'; the flake worsens on faster machines.
- Root cause
- The click races the CSS transition; Playwright's actionability check occasionally passes mid-transition.
- Fix
- Disable animations in the test environment with prefers-reduced-motion or a global CSS override that sets transition-duration: 0s.
Practice this concept in a real QA interview
Run a live mock with our AI Interview Coach, tune your resume with the ATS Resume Reviewer, and screen live listings on the QA Jobs Radar.