ReproSeed

Some bugs need a slow response and a click at the wrong moment. They reach a developer as a long list of steps, most of them irrelevant, and they fail only some of the time. Cutting the list down by hand means replaying it again and again, and a fix shipped without a test that failed first proves little.

ReproSeed takes a browser scenario that fails and a check that defines the failure, and does five things in order:

1. 01ReproduceReplay the whole scenario from a clean state; three baseline runs must fail the same way.
2. 02MinimizeDelta debugging removes groups of steps, then single steps, keeping only subsets that still fail the same way.
3. 03ProveTry deleting each remaining step once more; none may still reproduce the failure.
4. 04VerifyFive fresh replays of the result, with no cached answers, must all fail the same way.
5. 05ExportWrite a Playwright test that asserts the intended behaviour, so it fails until the bug is fixed.

Scope. It runs only against authorized loopback test applications, not arbitrary production websites. The app must run on the same machine and answer ReproSeed's identity and reset endpoints, as the two demo apps in the repository do.

A scenario is a JSON file: ordered steps (goto, click, fill, wait and delay), the steps each one depends on, and an oracle that defines the failure. The duplicate-submit scenario has nine.

  "steps": [
    { "id": "open", "kind": "goto", "path": "/", "dependsOn": [] },
    …
    {
      "id": "slow",
      "kind": "delay",
      "urlIncludes": "/api/records",
      "method": "POST",
      "ms": 2000,
      "dependsOn": []
    },
    { "id": "submit-1", "kind": "click", "selector": "#submit", "dependsOn": ["open", "email"] },
    { "id": "gap", "kind": "wait", "ms": 800, "dependsOn": [] },
    { "id": "submit-2", "kind": "click", "selector": "#submit", "dependsOn": ["open", "email"] }
  ],
  "oracle": {
    "id": "record-count",
    "request": { "method": "GET", "path": "/api/records" },
    "match": { "field": "email", "value": "a@example.com" },
    "expect": 1
  },

One replay

1. Check that the app on the loopback origin answers with the expected app id, then reset it with the credential from the environment. The record list must come back empty.
2. Open a fresh browser context. Requests to any other origin are blocked.
3. Run the steps. A delay step lets the request reach the server and holds only the response, the way a slow network would: Playwright fetches the response, waits, then delivers it.
4. Wait for the page's requests to settle, then run the oracle.

The oracle returns a signature, not a yes or no: DUPLICATE with the email, the 2 records seen and the 1 record expected. Only an exact match counts as the bug. A timeout, a runner error or a different failure, such as 0 records, is something else. Before the search starts, three baseline replays of the full scenario must agree; otherwise the run stops as flaky or not reproducible.

Kept to local test apps

It runs only against authorized loopback test applications, not arbitrary production websites.

The search is delta debugging: split the steps into chunks, try each chunk and each complement, and split finer when nothing smaller fails the same way.

A subset missing a step that another step depends on is rejected before any browser starts. A new subset is replayed twice, and a disagreement counts as inconclusive; a subset tried before is answered from a cache that keeps only results whose replays agreed. A budget bounds the whole run: 60 replays and 5 minutes for this scenario. Delta debugging can stop early, so the search ends with a proof pass: each remaining step is deleted on its own, and if a deletion still reproduces the failure, the step goes and the pass starts again.

1-minimal under single deletion of this ordered scenario, given the recorded probes. This is not a proof of the globally shortest reproduction.

In plain terms: no single remaining step can be deleted. That is a narrower claim than shortest. The search never reorders steps, never tries a shorter delay and never substitutes a different step, so a shorter reproduction may exist. Steps kept only because others depend on them, open and email here, have not been shown to cause the bug.

In the recorded run the search removed preview, note and decoy-wait and kept 6 steps, using 36 search replays and 11 cache hits. See every candidate in the explorer.

ReproSeed writes a test only for a verified run: a known baseline failure, kept steps in their original order with their dependencies, and five final replays that all reproduced it. An unstable, incomplete or over-budget run writes its report and no test.

The test asserts the intended behaviour, 1 record, not the 2 records the bug creates. It embeds the 6 kept steps with their selectors, values and delay, resets the app first, refuses a non-loopback origin before sending the credential, runs with retries off and does not import ReproSeed.

$ node dist/cli.js minimize examples/duplicate-submit/scenario.json --report artifacts/report.json --spec artifacts/minimized.spec.ts --force
Checking http://127.0.0.1:4010 for app duplicate-submit.
Minimizing 9 steps. This runs in Chromium and can take several minutes.
Status: verified-one-minimal
Result: 1-minimal under single deletion of this ordered scenario, given the recorded probes. This is not a proof of the globally shortest reproduction.
Original steps: 9
Kept steps: 6
Removed steps: preview, note, decoy-wait
Failure: DUPLICATE (email=a@example.com, actual=2, expect=1)
Replay executions: 44
  baseline: 3
  search: 36
  verify: 5
Candidate evaluations: 55
Cache hits: 11
Elapsed: 80252ms
Report: …/artifacts/report.json
Test: …/artifacts/minimized.spec.ts
The regression test asserts the intended record count. This command does not run it.

196test.describe.configure({ retries: 0, timeout: 20000 });
203  const config = readRuntimeConfig(process.env);
204  await confirmIdentity(config.origin, config.appId);
206  await resetAuthorized(config, emptyListPath);
207  const context = await browser.newContext();
212    removeGuard = await installOriginGuard(context, config.origin);
216      for (const step of scenario.steps) {
233    const records = await readRecords(config.origin, scenario.oracle.path);
234    const actual = countMatching(records, scenario.oracle.email);
238    expect(actual).toBe(1);

The command does not run the test it writes. Run with Playwright, the same file fails against the buggy page and passes against the repository's test-only fixed page.

Expected: 1
Received: 2
1 failed

1 passed (2.6s)

Playwright output, result lines Recorded output

$ npx playwright test --config test/export/playwright.config.ts
Running 1 test using 1 worker
[1/1] artifacts/minimized.spec.ts:198:1 › minimized scenario preserves the intended record count
1) artifacts/minimized.spec.ts:198:1 › minimized scenario preserves the intended record count
Error: expect(received).toBe(expected) // Object.is equality
Expected: 1
Received: 2
1 failed

$ npx playwright test --config test/export/playwright.config.ts
Running 1 test using 1 worker
[1/1] artifacts/minimized.spec.ts:198:1 › minimized scenario preserves the intended record count
1 passed (2.6s)

 let locked = false;
+Added: let pending = false;
 …
-Removed: if (locked) {
+Added: if (locked || pending) {
 …
 lastSubmitAt = now;
+Added: pending = true;
 …
-Removed: });
+Added: }).finally(() => {
+Added:   pending = false;
+Added: });

The same engine, a second bug

Stale search: a slow response to an earlier search arrives last and overwrites the results of the current search. It uses the same minimizer, runner, exporter and command with a different oracle, the text of one element. The minimizer has no search-specific code; it compares failure signatures. A run that never searches for beta shows “Results for alpha”, a different failure, so it does not count as the bug.

Steps

10 → 6

Removed

preview, note, decoy-wait and gap

Browser replays

52

Cache hits

14

Elapsed

30.1 s

One recorded run Recorded output · The README records a measured run of 52 replays, 14 cache hits and about 30 seconds Verified in source

Expected: "Results for beta"
Received: "Results for alpha (outdated)"
1 failed

1 passed (1.4s)