What a flaky test actually is
A flaky test is an automated test that passes and fails under identical code and circumstances. The commit did not change, the scenario did not change, the input did not change, and yet the result did. That is the whole definition, and it explains why one red run tells you so little: the same code would probably have passed a minute earlier.
The distinction that matters most is with a consistently failing test. A test that fails every time is deterministic, and a deterministic failure is a fact you can read — either the application broke, or the expectation no longer matches the intended behaviour. Both are worth acting on immediately. A test that fails one run in ten is a probability, and a probability has to be measured before it can be explained.
There is a third case worth naming, because it absorbs the most team time: a test that fails only on a colleague's machine, or only in the overnight scheduled run. That is not a separate category. It is a flaky test whose variable has not been found yet, because the environment counts as part of the circumstances.
What a flaky test is not is proof that the application is broken, and it is not proof that the test is wrong. It is a signal that something in the chain — the application, the test, the data, the environment — varies between runs. Your job is to find which link, not to pick a side in advance.
Run the diagnostic sequence before you change anything
The fastest way to lose a week is to edit the test the moment it goes red. This sequence is tool-free — a terminal, a repetition flag and a notebook are enough — and it replaces guesswork with observation.
Reproduce it on purpose. Run the same test repeatedly against the same commit, twenty or fifty executions is usually enough to separate a genuine flake from a one-off, and record every run's context: timestamp, environment, parallel worker, which tests ran before it, the failure message, and whether an immediate rerun passed. You are converting a vague complaint into a rate and a pattern.
Isolate one variable at a time. Take the failing test and change exactly one circumstance between repeat runs: alone versus in the full suite, suite order versus shuffled order, one worker versus many, clean data versus leftover data, stubbed dependencies versus the real ones, identical browser and viewport and time zone. A variable that moves the failure rate is a cause. A variable that changes nothing is a distraction, and you can stop investigating it.
Classify the failure before you fix it. In practice, flaky failures fall into a small number of families: timing and async waits, shared state or order dependency, external dependencies, and environment variation. The family decides the fix. A fix aimed at the wrong family usually relocates the failure instead of removing it.
The four families and the stabilization step for each
Automated tests become flaky when something other than the code varies between runs. These are the four things that vary most often, and the signal that identifies each one.
Family | Typical signal | Stabilization step |
|---|---|---|
Timing and async waits | Passes locally, fails on a slower or busier CI machine | Replace fixed sleeps with explicit waits or event-driven assertions |
Shared state or order dependency | Passes alone, fails in the full suite or when the order is shuffled | Isolate the test with clean, unique data |
External dependencies | Timeouts, unexpected 5xx responses, content that differs by region | Stub the external service and assert on how your application handles the response |
Environment variation | Fails only on one machine, browser, viewport or runner image | Pin and record the environment, and wait on conditions instead of durations |
Timing and async waits. The classic symptom is a step that acts on an element before the page has finished rendering it, so the test passes on a fast laptop and fails on a loaded CI runner. The reflex is to raise the timeout, which makes the suite slower and leaves the failure ready to return under load. Replace the fixed sleep with an explicit wait or an event-driven assertion on the condition the next step actually depends on: the request has returned, the spinner is gone, the button is enabled, the row is present with the expected value. A sleep encodes a guess about duration; a condition encodes the requirement.
Shared state or order dependency. A test that passes alone but fails in the suite, or changes behaviour when the suite is shuffled, is usually reading or writing state another test also uses — the same login account, the same cart, a database row left behind, a flag cached in local storage. Isolate the test with clean data: unique fixtures per test, a reset account or storage, and no reliance on another test's side effect. Order dependency is a property of the suite, so it will never appear while you debug that one test in isolation.
External dependencies. Failures that arrive as timeouts, unexpected error responses or region-specific content usually originate outside your application: a payment sandbox, an email provider, a hosted search index, a third-party API with its own rate limits and maintenance windows. Stub the external service at the boundary and assert on how your application handles a defined response, including the failure responses. Keep one small scheduled test against the real integration so the stub cannot drift silently away from the service it imitates.
Environment variation. Machine speed, browser version, viewport, locale, time zone, container image and the number of parallel workers can all change what a test observes without a line of code changing. Pin the environment and record it alongside the result — browser and version, viewport, locale, time zone, image tag, worker count — and remove the test's dependence on machine speed by waiting on conditions rather than durations. Anything you cannot pin, log, because it is the first thing you will want to compare when the next failure appears.
Is the test wrong, or is the application broken?
The failure is evidence about one of them, and the order of investigation decides whether you spend the afternoon on the test or on a real defect.
Does the failure cluster around specific data? A failure that appears with one cart total, one discount code, one currency or one empty field is probably deterministic. You have found an input that breaks the application, and the intermittent test was the messenger. Reproduce it with that value before you touch the test.
Is the wrong thing a value or a missing element? A missing element on a page that was still loading is usually timing. A value that is present but incorrect — the order total, the confirmation number, the balance after a refund — is the application making a wrong claim, and adding a wait only makes the wrong value arrive on time.
Did the failures begin at a deploy or a configuration change? If the first red run coincides with a release and stops when the release is rolled back, the test is reporting a regression. If it began when the CI runner image changed, it is environment variation.
Does the failure involve a duplicated action? Two submissions, a double charge, a record created twice — that is a concurrency problem in the application, and the intermittency is the signature of the bug rather than a defect in the test.
The rule underneath those questions is simple: editing a test is a claim that the expectation was wrong. Until you can explain why the failure happened, a change to the test hides the signal instead of removing it.
Where BuniOD fits after the method
Everything above works with any test runner, and the diagnostic sequence is what actually makes a suite trustworthy. Tools help at the edges, and those edges are exactly where recurring failures are found.
BuniOD is an AI test automation tool for web applications, and three of its stated behaviours map onto this article. When an element moves, it retries the stored locators, then element fingerprints, then AI matching before a step fails, and saves the new location after a passing run so the next run starts from the corrected locator — a practical answer to the locator share of the timing family. When a run does fail, the failure names the exact step or check that failed and explains in plain language why, framed as a choice between a bug in the application and a scenario that needs updating rather than a stack trace. That explanation sits next to the same run's step screenshot and logs, so the evidence needed for triage arrives with the failure instead of requiring a manual reproduction, and failed runs are not retried automatically, so a red result is the real result of that run.
Two qualifications come from the product's own framing and matter here. Self-healing element matching applies where the element can still be identified after the change; when the intended flow itself changes, the scenario needs an updated description and a human review. And none of this promises a specific reduction in failure rate. Whether flakiness falls depends on which family your failures belong to, which is why the diagnosis comes first.
Making stability a standing practice
One repaired test is a result; a suite that stays trustworthy is a process. Four habits carry most of the weight.
Set a completion standard before you trust a test again. A workable one: the test must pass a run of consecutive executions — twenty, for example — across the environments it actually runs in, with no code change in between. Without a number, "it seems fixed now" is the same guess that produced the flake.
Quarantine instead of silencing. Move a known-flaky test into a quarantined set that still runs and still reports but does not block merges, and time-box how long it may stay there. The point is to keep the signal while removing the noise, not to delete the test.
Cap retries and count them. Automatic retries are the most common way a flaky suite becomes invisible: the build turns green, the instability remains, and a genuine regression that appears once in twenty runs becomes indistinguishable from a flake. If you retry, cap the attempts, record every retry as a failure of the first attempt, and track the retry rate per test.
Fix by family, then re-measure. A per-instance patch — a longer wait here, a looser assertion there — moves the failure to the next test in the same family. Apply the family's stabilization step across the suite, then compare the failure rate before and after over the same number of runs. That comparison is the evidence that the change worked, and it does not require anyone to promise a percentage.
The trade-offs deserve to be stated plainly. Stubbing external services reduces what the suite proves about real integrations, so pair it with a scheduled test against the live service. Pinning the environment reduces the variety the suite exercises, so treat it as a deliberate decision to stabilise and to limit. Both are acceptable as long as the choice is made on purpose.
