Skip to content

ADR 0021: Benchmark methodology

Status: Proposed · 2026-11-17 (milestone M6 · Benchmark and hardening; decision tracked in #120, #123)

Context

Claims about cost and quality need evidence that others can reproduce.

Decision

A fixed public corpus, a baseline of the same agent without Wayloop under the same budget, repeated runs with reported spread, per-phase cost attribution, and publication of harness and raw data. Results never name other products.

Consequences

Claims are verifiable. Benchmark runs cost money and are scheduled, not run on every pull request.