{"slug":"flaky-test-review","name":"Flaky Test Review","version":"1.0.0","updated_at":"2026-10-26T21:01:00.523Z","use_when":"Reviews ONE test file for the five causes that make a test pass on one run and fail on the next, and lists only those, one line each with a fixed code, the test, why it flips and the fix - or exactly No findings. It catches a fixed sleep used to wait for background work, an expectation tied to today's date or the current time, tests that share state and only pass in one order, random values with no fixed seed, and a call to a real network address. It leaves out the safe look-alikes, such as a fake clock set before the call, a random generator built from a fixed seed, a mocked network module and fake timers moved by hand. Use to find why a test fails once in a while in CI, to review a new test file before merging, or to harden a suite that people re-run until it goes green.","not_for":"Tests that cannot fail, test speed, coverage, automatic reruns in CI and fixtures kept in other files. It reads one pasted test file and judges only what is written there, not the code under test.","languages":["en"],"tags":["flaky-tests","testing","ci","determinism","pytest","jest"],"category":"code","category_url":"https://aiskills402.com/categories/code","keywords":["fixed sleep","fake clock","random values","mocked network"],"faq":[{"q":"Frozen time set in a setup hook: does it count as a clock problem?","a":"No. A fake clock set before the call is the safe form, and so are fake timers that the test moves by hand. Only an expectation that reads the real date or time, with nothing earlier in the file to freeze it, gets a line."},{"q":"Where do the five causes come from?","a":"They are the skill's own list. The Google Testing Blog post names random numbers and absolute timing among the causes, but our fetch returned only the reader replies, so the text claims no more than that and gives no rate or number."},{"q":"Which files went through the trial?","a":"A pytest file with a list shared across tests and a three-second pause, a Jest file with a month in the expected text and a real rates address, and a clean Jest file with mocks and a fixed seed. Sonnet and Haiku each read all three once."},{"q":"What does a stable file return?","a":"Only the sentence No findings. A cause is reported on one line: the code, the test, the reason it flips and the repair."}],"examples":[{"lang":"en","model":"claude-sonnet-5-5","input_excerpt":"import random\nimport time\n\nfrom app.jobs import start_export, export_done\nfrom app.reports import build_report\nfrom app.lottery import draw\n\nitems = []\n\n\ndef test_adds_first_item():\n    items.append(\"a\")\n    assert items == [\"a\"]\n\n\ndef test_adds_second_item():\n    items.append(\"b\")\n    assert items == [\"a\", \"b\"]\n\n\ndef test_export_finishes():\n    job = start_export()\n    time.sleep(3)…","output_excerpt":"FIXED-SLEEP — test_export_finishes, `time.sleep(3)` — on a slow runner three seconds may not be enough, and on a fast one it only wastes time — poll `export_done(job)` in a loop with a timeout instead of sleeping…"}],"page_url":"https://aiskills402.com/skills/flaky-test-review","markdown_url":"https://aiskills402.com/skills/flaky-test-review.md","image_url":"https://cdn.aiskills402.com/og/skills/flaky-test-review/ca9533f9.png","related_url":"https://api.aiskills402.com/v1/skills/flaky-test-review/related","purchases_count":null,"tested":{"date":"2026-10-26","strong":{"model":"claude-sonnet-5-5 (Claude Code alias \"sonnet\")","verdict":"Right on all 3 cases, read by hand. On the pytest file it found the three-second pause and the module-level list that the second test relies on, and left out the seeded generator and the fake clock fixture. On the Jest file it found the expected month tied to today and the real rates address, and left out the seeded winner test. On the clean Jest file, with its frozen clock, mocked module, fixed seed and hand-moved timers, it answered exactly No findings."},"weak":{"model":"claude-haiku-5-5 (Claude Code alias \"haiku\")","verdict":"Right on all 3 cases, read by hand. It matched the strong model on both planted files, with longer reasons that add the case of running a test alone or in random order, and kept every safe look-alike out. On the clean file it answered exactly No findings. One answer was indented by four spaces, which the check accepts."},"note":"Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three inputs were written by us; each planted one carries a safe look-alike and neither planted case has a comment pointing at the defect. The public source was read only through its reader replies, not the article body, so the skill claims little from it; the five codes are our own list. There are no owner-measured facts.","report_url":null},"price_usd":"0.01","price_micro":10000,"size_bytes":5868,"sha256":"694db5737df5f70a66c724954574c1ac38e069393d6618af782c84ed03af0acb","outline":["What the public source says, as read on 26 October 2026","The answer","The codes","Rules","Work in this order","Short example"],"license":{"summary":"Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing","holder":"Georgi Kalchev, aiskills402.com","url":"https://aiskills402.com/docs#license"},"buy_url":"https://api.aiskills402.com/v1/skills/flaky-test-review/file","redownload_url_template":"https://api.aiskills402.com/v1/purchases/{token}","mcp_tool":null,"payment":{"protocol":"x402","scheme":"exact","asset":"USDC","selling":true,"network":"base","network_caip2":"eip155:8453","pay_to":"0x8e37022edcf0f21cf3c9f93fee9d4d32519f36f4","facilitator":"cdp"},"seo_title":"Flaky Test Review for Unstable CI Tests","seo_description":"Reviews one test file for fixed sleeps, real-clock dates, shared state, unseeded random values and live network calls. Pay $0.01 once, in USDC.","versions":[{"version":"1.0.0","date":"2026-10-26","changelog":"# Changelog\n\n## 1.0.0 — 2026-10-26\n\nFirst release, written in the second light-test batch of 26 October 2026 (three cases, price $0.01). The idea is ours; no foreign skill was opened, facts come from a public testing blog post (see notes/ideas.md). Output: see the skill description.\n\nTest: three cases written by hand (two planted, each with a safe look-alike, one clean control), one run on Claude Sonnet and one on Claude Haiku, no run without the skill, no iteration on the skill rules after the runs. `test/control.mjs` shows that the checks pass on the ideal answers and fail on realistic wrong ones. See `test-summary.json` for the verdicts read by hand.\n"}]}