{"slug":"ab-test-readout-check","name":"A/B Test Readout Check","version":"1.0.0","updated_at":"2026-10-19T06:01:01.347Z","use_when":"Checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a \"win\", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings. It catches stopping the moment a result looks significant, many metrics or variants judged without correction, a split of users that differs from the planned split by more than chance allows (the sample ratio check, calculated step by step), a sample too small for the effect the team wants to detect, and a relative lift reported without the absolute numbers behind it. It leaves out the safe look-alikes, such as a pre-registered single primary metric, guardrail metrics that only monitor, and a split of 50.2 to 49.8 on a small sample. Use before a test result is shipped, when an analyst or an agent writes a readout, or to review a result someone else announced.","not_for":"Designing an experiment before it runs, recomputing p-values or intervals, choosing a statistical test, Bayesian readouts, segment analysis, or deciding whether to ship. It reads one pasted readout and judges only how its result was reached.","languages":["en"],"tags":["ab-testing","experimentation","statistics","sample-ratio","data-review","product-analytics"],"category":"data","category_url":"https://aiskills402.com/categories/data","keywords":["sample ratio","relative lift","absolute numbers","small sample"],"faq":[{"q":"When does a split of users count as a mismatch?","a":"It works through the arithmetic in the answer: total users, expected count per group from the planned split, squared difference over expected for each group, then the sum against the limit 10.83 for two groups. A 20,000 against 21,000 split on a planned 50 to 50 is reported; 502 against 498 is not."},{"q":"Does it demand a power calculation from every readout?","a":"No. The sample size is checked only when the readout itself names the smallest change the team hoped to find, using a rule of thumb that the skill shows with a worked example. A readout that gives no minimum effect is not judged underpowered, because the skill never invents a number the text does not contain."},{"q":"What does it leave out on purpose?","a":"Guardrail metrics shown only for monitoring, a single pre-registered primary metric beside other listed numbers, a relative lift printed next to both absolute rates, and a small-sample split that passes the check. These are the look-alikes in our three test cases, and none of them gets a line."},{"q":"How was it tested?","a":"Our trial used an early stop with an uneven split, a six-metric table with a lift in percent only, and a clean pre-registered readout. Claude Sonnet and Claude Haiku each answered once per readout. Sonnet matched all three; Haiku matched the two planted ones and broke the fixed clean sentence with an extra paragraph."}],"examples":[{"lang":"en","model":"claude-sonnet-5-5","input_excerpt":"Readout: onboarding redesign (variant B against control A)\n\nRan for the full 14 days as planned. Users were split 50 to 50: A got 4,012 users and B got 3,988 users.\n\nMetric                      Change for B     p-value\nAdd-to-cart rate            +14 percent      0.03\nCheckout start rate         +2 percent       0.41\nPurchase rate               +1 percent       0.68…","output_excerpt":"MULTIPLE-COMPARISONS — add-to-cart rate \"clear win\" among six metrics — six metrics were tested (add-to-cart, checkout start, purchase, average order value, return visit, support tickets), no primary metric is named, and the win is declared at the plain p = 0.03 < 0.05; a Bonferroni threshold for 6 metrics is 0.05 / 6 ≈ 0.0083, which 0.03 does not meet — name the primary metric as planned before…"}],"page_url":"https://aiskills402.com/skills/ab-test-readout-check","markdown_url":"https://aiskills402.com/skills/ab-test-readout-check.md","image_url":"https://cdn.aiskills402.com/og/skills/ab-test-readout-check/055a9f13.png","related_url":"https://api.aiskills402.com/v1/skills/ab-test-readout-check/related","purchases_count":null,"tested":{"date":"2026-10-19","strong":{"model":"claude-sonnet-5-5 (Claude Code alias \"sonnet\")","verdict":"Right on all 3 cases, read by hand. On a checkout-button readout it gave the early stop on day 9 of 14 and the uneven split of 48,300 against 50,100, with the sample ratio arithmetic done correctly to 32.93, and it left out the three guardrail metrics. On a six-metric table it gave the uncorrected win at p of 0.03 and the lift reported only in percent, and left out the 4,012 against 3,988 split. On a pre-registered signup readout it answered exactly No findings."},"weak":{"model":"claude-haiku-5-5 (Claude Code alias \"haiku\")","verdict":"Right on 2 of 3 cases, read by hand. Both planted readouts got exactly the expected lines, with the same correct arithmetic (32.93 against the limit 10.83) and the safe look-alikes left out. On the clean signup readout it wrote No findings. and then added a paragraph that the variant had 39,880 users against a planned minimum of 40,000. That is a true observation about a wrinkle we did not intend in our own test text, but it breaks the fixed sentence, so it is recorded as a wrong answer."},"note":"Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three readouts were written by us; each planted one carries a safe look-alike (guardrail metrics, a 4,012 against 3,988 split, a relative lift printed next to absolute rates in the clean case), and neither planted case has a comment pointing at the defect. The clean readout accidentally says the planned sample was at least 40,000 per group while the variant has 39,880; we did not edit it after the run. Facts come from public statistics references, read on 19 October 2026; there are no owner-measured facts.","report_url":null},"price_usd":"0.01","price_micro":10000,"size_bytes":9307,"sha256":"145ce399f8b5cd12e4d48092b23b11a604379ef142906c6dbc20e42115350a8c","outline":["The answer","The codes","The sample ratio check, step by step","The sample size check","How to read the readout","Rules","Work in this order","Short example"],"license":{"summary":"Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing","holder":"Georgi Kalchev, aiskills402.com","url":"https://aiskills402.com/docs#license"},"buy_url":"https://api.aiskills402.com/v1/skills/ab-test-readout-check/file","redownload_url_template":"https://api.aiskills402.com/v1/purchases/{token}","mcp_tool":null,"payment":{"protocol":"x402","scheme":"exact","asset":"USDC","selling":true,"network":"base","network_caip2":"eip155:8453","pay_to":"0x8e37022edcf0f21cf3c9f93fee9d4d32519f36f4","facilitator":"cdp"},"seo_title":"A/B Test Readout Check: Peeking, SRM, Lift","seo_description":"Checks a written A/B result for early stopping, many metrics, a split that fails the sample ratio check and lift without numbers. Pay $0.01 once, in USDC.","versions":[{"version":"1.0.0","date":"2026-10-19","changelog":"# Changelog\n\n## 1.0.0 — 2026-10-19\n\nFirst release, written in the light-test trial of 19 October 2026 (three cases, price $0.01). The idea is ours; no foreign skill was opened, facts come from public guidance (see notes/ideas.md). Output: one line per finding with a fixed code, or exactly `No findings.`\n\nTest: three cases written by hand (two planted, each with a safe look-alike, one clean control), one run on Claude Sonnet and one on Claude Haiku, no run without the skill, no iteration on the skill text after the runs. `test/control.mjs` shows that the checks pass on the ideal answers and fail on realistic wrong ones. See `test-summary.json` for the verdicts read by hand.\n"}]}