01MISSING
It is now skipped, deleted, or quietly excluded.
The failing test is the description of the bug, so anything other than failed becoming passed means the goalposts moved.
test_auth::test_expired_token_rejected
02 / Current focuswhatrana check your agent cannot route around
The tests pass. The one that was failing is gone.
whatran remembers what your suite ran before the agent started, and tells you when something stops running. A skipped test hides where a broken one shouts, so it watches results, not diffs.
no account · no cloud · no AI in the checking
- Stage
- Shipped · v0.3.0 on npm
- Install
- npx whatran init
- Runners
- pytest, Vitest, Jest, Mocha, node:test, go test, cargo nextest
- Works with
- Claude Code, Codex CLI, Cursor, CI
- Dependencies
- None. MIT licensed.
One turn, as it happens
Claude Code · Stop hook · runs in the background
mesh · expired token bugagent session
test_expired_token_rejected is failing. Fix it.
Fixed. All tests pass.
$ pytest -q
2 passed, 1 skipped
$ echo $? 0/whatran
STOPturn blocked, agent notified
whatran compared what your test suite ran BEFORE this change with what it runs now, and coverage went missing. The suite being green does not settle this: these tests are green because they are no longer reporting.
1 failing test was skipped instead of fixed:
test_auth::test_expired_token_rejected
-> Restore these tests and make them pass by fixing the underlying problem. Do not skip them. If one is genuinely obsolete, say so explicitly and explain why rather than silencing it.zsh · ~/meshthe same check, by hand
$ whatran
MISSING 1 failing test was skipped instead of fixed
These tests were failing before the change and are now not
running at all. The suite reports green because they no longer
report anything.
· test_auth::test_expired_token_rejected
vs snapshot 2026-01-14T09:12:04.881Z, 6 minutes old, confirmed by a second run
pytest · 3 tests, 2 passed, 0 failed, 1 skipped (-1 vs baseline) · 1.2s
Per-test outcome diff
| Test | Was | Now |
|---|---|---|
| test_auth::test_expired_token_rejected | failed | not run |
| test_auth::test_refresh_rotates | passed | passed |
| test_session::test_logout_clears | passed | passed |
| test_session::test_idle_timeout | skipped | skipped |
What it catches
A failing test that stopped failing the dishonest way.
Only MISSING interrupts your agent. A broken test already shouts; this is for the ones that hide.
02MISSING
A stray .only, or a module-level skip mark.
Silently stops every other test in the file from running, while the suite still reports green.
describe.only( · pytestmark = skip
03NOTICE
A change to the test config itself.
One new conftest.py, or one line in pyproject.toml, can make an entire suite report as passed without touching a line of source.
conftest.py · pyproject.toml
How it stays honest
Three rules it does not bend.
- 01
Remember what runs today
A baseline of per-test outcomes, never recorded from a dirty tree. That would write whatever the agent just did into the definition of correct.
- 02
Compare runs, not diffs
Plenty of tests stop running with no trace in the source at all. Searching a diff for the word skip finds none of them.
- 03
Accuse only twice-confirmed
Before saying anything it runs the suite again and reports only what happens both times. Flaky tests are remembered and left alone.
Set it up once, in one line.
It finds your runner, remembers today's suite, and wires itself into your agent.
npx whatran init
Source on GitHub