ADR 0021: Benchmark methodology¶
Status: Proposed · 2026-11-17 (milestone M6 · Benchmark and hardening; decision tracked in #120, #123)
Context¶
Claims about cost and quality need evidence that others can reproduce.
Decision¶
A fixed public corpus, a baseline of the same agent without Wayloop under the same budget, repeated runs with reported spread, per-phase cost attribution, and publication of harness and raw data. Results never name other products.
Consequences¶
Claims are verifiable. Benchmark runs cost money and are scheduled, not run on every pull request.