Flaky Tests Are a Reliability Problem: Building a Quarantine Strategy That Works - SysRoot

The Pipeline That Cried Wolf

Most teams treat flaky tests as an annoyance, something to re-run until green. That framing is the problem. A flaky test is not a minor inconvenience; it is a slow corrosion of the single most valuable property your CI/CD pipeline has: signal you can trust.

When a test fails intermittently and developers learn that hitting "re-run" usually fixes it, they stop reading failures. Once they stop reading failures, real regressions sail straight through to main. The pipeline is technically still running, the dashboards are green often enough, but the pipeline has stopped doing its job. You have replaced a quality gate with a coin flip.

This article is about one specific lever: a disciplined flaky-test quarantine strategy. Done well, it restores trust in green builds, keeps deploys flowing, and surfaces flaky tests as a tracked, owned class of work instead of an ambient cost everyone absorbs silently. Done badly, it becomes a junk drawer where failing tests go to be forgotten.

Why Flakiness Quietly Destroys Velocity

The cost of flaky tests is rarely a single dramatic event. It is death by a thousand re-runs.

Consider a mid-sized team running a 14-minute test suite across pull requests. Suppose 3% of CI runs fail due to flakiness rather than a genuine defect. With 60 merges a day and the typical pattern of one or two re-runs per flaky failure, you are burning compute, blocking merges, and forcing context switches. But the compute waste is the cheap part.

The expensive part is behavioral:

• Developers default to re-running rather than investigating, so genuine failures get re-run too.
• Reviewers stop blocking merges on red CI because red has become noise.
• On-call engineers lose confidence that "all tests passed" means anything during an incident.
• New hires learn within their first week that the pipeline lies, and they calibrate accordingly.

Once trust is gone, recovering it is far harder than the original engineering fix. That is why flakiness belongs in the reliability conversation, not the test-hygiene backlog.

What Actually Causes Flakiness

Before you can quarantine intelligently, you need to recognize the common failure modes. Quarantine without classification just hides root causes.

Timing and ordering assumptions

The largest category. Tests that assume an async operation completes within a fixed sleep, tests that depend on map/dictionary iteration order, or tests that pass only when run after a sibling that happened to seed shared state. These fail when the CI runner is under load or when test order changes.

Shared mutable state

Tests that write to the same database row, the same Redis key, the same temp file, or the same global singleton. They pass in isolation and fail under parallelism. As you scale runners horizontally to keep suites fast, this class explodes.

External dependencies

Tests that hit real network endpoints, third-party APIs, or DNS. These fail for reasons completely unrelated to your code. They are not testing your logic; they are testing someone else's uptime.

Resource contention on the runner

Under memory pressure or CPU throttling, a test that normally finishes in 200ms takes 4 seconds and trips a timeout. This is why flakiness often correlates with busy parts of the day or with shared, oversubscribed CI infrastructure.

Nondeterministic test data

Randomized inputs without a fixed seed, timestamps that cross day boundaries, or locale/timezone assumptions that break on a differently-configured runner.

Notice that some of these are test bugs and some are infrastructure problems. A quarantine process that treats them all identically will mask real infrastructure rot.

The Core Idea: Quarantine as a Tracked State, Not a Trash Can

The purpose of quarantine is to remove a noisy test from the blocking path while keeping it visible, owned, and on a deadline. A quarantined test must still run; it just must not block merges. Its results are recorded so you can measure whether it is genuinely flaky or actually catching a real intermittent bug.

The failure mode to avoid: a @flaky annotation that silently skips a test forever. That is not quarantine, that is deletion with extra steps and false comfort. Within six months you will have hundreds of "quarantined" tests, no idea which ones still matter, and a coverage map that lies to you.

A real quarantine has four properties:

1. The test still executes on every relevant run.
2. Its failures do not block the merge.
3. Every quarantined test has an owner and a tracking ticket.
4. There is an expiry: a test cannot stay quarantined indefinitely.

Building the Quarantine Pipeline: Step by Step

Here is a concrete, tool-agnostic process you can implement in any CI system (GitHub Actions, GitLab CI, Buildkite, Jenkins).

1. Detect flakiness automatically

Do not rely on humans to label flaky tests. Humans label the loud ones and ignore the rest. Instead, record per-test results across runs and flag any test that passes and fails on the same commit SHA. Same code, different result, is the cleanest signal of flakiness available.

Many CI vendors offer flaky-test detection natively. If yours does not, store test results (JUnit XML is the lingua franca) in a small database keyed by test name plus commit. A test that has both a pass and a fail for an identical SHA is flaky by definition.

2. Split your suite into blocking and non-blocking stages

Structure the pipeline so there are two test stages:

• Blocking suite: stable tests. A failure here blocks the merge, full stop.
• Quarantine suite: known-flaky tests. These run, results are recorded, but a failure here does not block the merge.

The mechanism is usually a tag or a manifest file (quarantine.txt) listing test identifiers. Your test runner reads the manifest and routes accordingly. Keep the manifest in version control so quarantine changes go through code review.

3. Require an owner and a ticket per entry

Enforce this in CI. When the quarantine.txt file changes, a lint check verifies each new entry has a corresponding ticket reference and an owner. No ticket, no merge. This single rule prevents the junk-drawer outcome more than anything else.

4. Set an expiry

Attach a date to each entry. A weekly job scans the manifest and fails (or files an escalation) for any quarantined test older than, say, 30 days. The test must either be fixed and returned to the blocking suite, or formally deleted with justification. Quarantine is a hospital, not a hospice.

5. Surface quarantine health as a metric

Track the count of quarantined tests over time. A growing number means flakiness is outpacing your ability to fix it, which is itself a reliability signal worth a conversation. Stable or shrinking is healthy.

A Realistic Scenario

A Series-A SaaS company runs a Django monolith with about 4,000 tests. After scaling CI from 1 to 8 parallel runners to cut suite time from 22 to 4 minutes, failure rates spiked. Developers blamed "CI being broken" and started re-running everything.

The real story: parallelism exposed roughly 40 tests that shared a database state through a module-level fixture. In serial execution they ran in a predictable order and passed. Under parallelism, two workers raced on the same fixture.

The team's first instinct was to roll back to serial execution. That would have meant accepting a 22-minute suite indefinitely, taxing every single merge to paper over 1% of tests.

Instead they did this:

1. Enabled same-SHA flaky detection, which surfaced the exact 40 tests within two days.
2. Moved those 40 into a quarantine suite so merges flowed again at 4 minutes.
3. Filed one ticket per affected module, assigned to the owning teams.
4. Fixed the root cause: replaced the module-level shared fixture with per-test transactional isolation.

Within three weeks, 38 of the 40 returned to the blocking suite. The remaining two were genuinely testing a third-party integration and were rewritten to mock the external call. The quarantine count dropped to zero, and crucially, trust in green builds returned. Developers started reading failures again because failures meant something.

The key insight: quarantine bought them speed while they fixed the root cause. It did not become the permanent solution.

Trade-offs You Should Decide Deliberately

Hiding real bugs

The biggest risk. A test marked flaky might actually be catching a genuine race condition in your production code, not a test bug. This is why same-SHA detection matters and why every quarantine entry needs investigation, not just suppression. If a "flaky" test fails 50% of the time, your code probably has a real concurrency bug that will eventually hit production.

Quarantine suite still costs compute

You are still running these tests, just not blocking on them. That is intentional, you need the data, but it is not free. Budget for it.

Manifest maintenance overhead

The ticket-and-expiry discipline takes effort. Teams that skip it get the junk drawer. The discipline is the strategy; without it you have nothing.

Retries are a complement, not a substitute

Automatic retries of individual tests can reduce noise, but unlimited retries hide everything. If you use retries, cap them at one or two, and feed retry-and-pass results into your flaky detection. A test that needs a retry to pass is flaky and belongs in the process, not silently rescued.

A Practical Checklist

Use this to assess or build your own quarantine strategy:

• Test results are recorded per test, per commit SHA, across runs.
• Flaky tests are detected automatically via same-SHA pass/fail divergence.
• The suite is split into a blocking stage and a non-blocking quarantine stage.
• Quarantined tests still execute on every run.
• Each quarantine entry requires an owner and a tracking ticket, enforced in CI.
• Each entry has an expiry date; stale entries trigger escalation.
• Test retries, if used, are capped and counted as flakiness signals.
• The total quarantine count is tracked as a visible reliability metric.
• Returning a test to blocking requires evidence it passes reliably, not just "it passed once."

The Bottom Line

Flaky tests are not a testing problem you delegate to whoever is least busy. They are a reliability problem that determines whether your entire CI/CD pipeline produces trustworthy signal. A green build that nobody believes is worse than no build at all, because it provides false confidence at the exact moment you need real confidence: shipping to production.

Quarantine is the operational tool that lets you keep velocity while you fix root causes, but only if it stays a tracked, time-boxed, owned state rather than a place tests go to die. Build the discipline into the pipeline itself, measure the count, and treat a growing quarantine as the warning it is. The goal is not zero failures. The goal is a pipeline where every red means something, and every green can be trusted.

---

Struggling with CI/CD Reliability?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with CI/CD Reliability, talk to the SysRoot team and we'll help you get it under control.