A timeout reported as a normal exit
Development review rev_20260904131435_mf6w identified a supervisor-state problem. The driving agent investigated it and added the timeout_reports_timed_out regression test.
EVIDENCE, WITH ITS LIMITS
Every number needs
a denominator.
A reproducible snapshot of the project’s own development reviews. Not an independent benchmark. Not a promise about your code.
SNAPSHOT / 2026-09-04 · Download JSON · SHA-256 sidecar
153recorded runs
139stored reports
570decision records
MOMM 1.17.3 release checks—not historical telemetry
Published 2026-10-06. Exact release revision 507e5f77 passed the Windows / macOS / Linux and Node 18 to 24 matrix. Its release workflow passed a genuine signed preview and apply before publishing the tag.
These public workflow results test release mechanics and deterministic fixtures. They do not prove that every reviewer account is available, that all providers answered a new project, or that MOMM finds every defect. Read this version’s notes →
Why the totals differ—and reconcile
Run logs record 366 successful reviewer responses. The detailed stored reports contain 343; the remaining 23 are represented by summary records only. The route table below uses stored reports, not all run logs.
The historical “other” bucket preserves old labels such as recommended; we do not invent retrospective rulings. 31 coalition or multiple-route decisions (26 applied, 5 rejected, 0 deferred, 0 other) are separate from single-route rows.
Route-level development record
| CLI route | Completed | Timeouts | Other failures | Median s | p90 s | Applied | Rejected | Deferred | Other | Acceptance |
|---|---|---|---|---|---|---|---|---|---|---|
| codex | 70 | 17 | 0 | 114.4 | 341.2 | 94 | 41 | 2 | 0 | 70% |
| claude | 21 | 2 | 0 | 37 | 274.5 | 26 | 37 | 0 | 8 | 41% |
| antigravity | 83 | 1 | 20 | 32.3 | 128.4 | 46 | 36 | 0 | 1 | 56% |
| copilot | 101 | 2 | 3 | 32.2 | 213.3 | 76 | 101 | 1 | 0 | 43% |
| grok | 68 | 25 | 2 | 29.8 | 328 | 34 | 31 | 5 | 0 | 52% |
| gemini | 0 | 0 | 6 | — | — | 0 | 0 | 0 | 0 | — |
Acceptance = applied ÷ (applied + rejected), according to this project’s governor. It is not precision against a labeled ground truth. Timing percentiles use the nearest-rank method on successful stored responses; failed routes are not hidden in the completed count.
Routes CSV · Markdown table · All attribution buckets · Decision records CSV · Run records CSV · Generated statistics
See the record, not just the headline
Historical snapshot 2026-09-04. All five charts are generated offline from the same sanitized download. No live account data is fetched.
Time to a usable review
What completed reviewers said
▰ ACCEPT▰ MODIFY▰ REJECT
Severity is a claim
Input size and observed time
When these runs happened
Which routes returned a usable review?
“External results” includes every stored non-self-excluded route result, even routes refused before spawning. It is not an estimate of paid calls. Self-exclusion is counted separately, never as a failure.
| Route | External results | Completed | Completion | Self-excluded | Failure breakdown |
|---|---|---|---|---|---|
| codex | 87 | 70 | 80.5% | 14 | timeout: 17 |
| claude | 23 | 21 | 91.3% | 81 | timeout: 2 |
| antigravity | 104 | 83 | 79.8% | 0 | error: 6 · timeout: 1 · invalid_output: 14 |
| copilot | 106 | 101 | 95.3% | 0 | authentication_required: 3 · timeout: 2 |
| grok | 95 | 68 | 71.6% | 0 | authentication_required: 1 · invalid_output: 1 · timeout: 25 |
| gemini | 6 | 0 | 0.0% | 0 | error: 2 · ineligible_tier: 4 |
“Other failures” in the timing table means every status except success, timeout and self-excluded. The breakdown above preserves the actual labels. Missing reports are not inferred.
What could improve for you—and what we have measured
MOMM adds an explicit challenge-and-verification step to AI-assisted work. The practical benefits to look for are additional defects identified, unsupported suggestions rejected, and a review history you can inspect. Their size depends on your tasks, reviewers and the governor’s tests.
| Potential benefit | Observation in this development snapshot | What it does not establish |
|---|---|---|
| Find changes worth making | 302 recorded applied decisions; the linked supervisor case includes a named regression test. | Not 302 unique bugs, independently verified fixes or a user defect-reduction percentage. |
| Filter unsupported advice | 251 recorded rejected decisions; a rejected CRITICAL claim remains inspectable. | Not a measured false-positive rate or a guarantee that the governor always rules correctly. |
| Keep an inspectable decision trail | 570 decision records across 153 logged runs; 139 detailed stored reports. 8 decisions are deferred and 9 retain historical labels. | Not a count of users, deployments, commercial outcomes or independent replications. |
No controlled AI-alone versus AI-plus-MOMM comparison is included in this snapshot. It cannot establish an average accuracy gain, time saving, cost saving or return on investment. The timing charts measure reviewer responses, not end-to-end developer productivity. Adding reviewers can add time and provider cost.
Snapshot date: 2026-09-04. Inspect the decision rows · Recompute the statistics · Inspect the concrete cases.
Measure the benefit on your own work
Use a held-out set of representative tasks and a pre-defined test oracle. Compare the same base model and settings with and without MOMM, vary run order, and repeat trials. Include failed routes, total elapsed time, provider costs, confirmed defects and regressions introduced. Report the sample size, uncertainty and evaluation method before claiming a general improvement.
A fix—and a claim we did not trust
Severity is a claim too
Follow-up review rev_20260904134630_bl2v included a CRITICAL claim that did not survive reproduction. The rejection and reasoning remain in the same public evidence export.
These are developer-recorded decisions about MOMM’s own code. The export preserves their evidence trail; it does not independently validate every historical claim.
Which bytes does a hash cover?
- Snapshot sidecar
- The exact downloaded
momm-evidence.jsonbytes. You can recompute this locally. stored_report_sha256- Original private report bytes before sanitization. Provenance only; it cannot validate the altered public object.
public_report_sha256- The sanitized report serialized as compact UTF-8 JSON with recursively sorted object keys; array order is retained. The renderer and CI recompute every one.
- Executable hashes
- New reports identify the actual installed dispatcher, updater and protocol file bytes. A version string alone is not evidence that two executions used identical code.
How this page stays in sync
One offline renderer reads the committed public snapshot and generates this page, its tables and downloadable data. CI compares the regenerated outputs byte for byte, checks the ledger’s embedded data, and verifies the public hashes. Refreshing the site does not import private ledgers or change the snapshot’s date.