Matched work
Compare the same coding tasks, tool contracts, context budgets and protected graders. Pin the application checkout and compiler. Record model, template, numerical-format and sampling differences so a result can be attributed within those limits.
Include edits, tool-result follow-ups, retries and longer conversations, and state which languages and applications the tasks cover. Component tests establish how individual parts behave. Assess the coding agent on complete tasks, checking the resulting application as well as its test output.
Correctness
Freeze inputs and grading rules before timing. Retain every attempt in the result record. Check accepted tokens and final model state when evaluating speculative decoding or context reuse.
Report the number of tasks attempted and passed alongside timing. Use protected tests where a generated change could otherwise satisfy a visible test without solving the task.
Timing boundaries
Complete-task time covers the agent’s work through the required checks. Report model-request timing separately, including queueing, time to first token and generation. State whether loading, preparation and tool execution are included in each measure.
The warm-cache benchmark uses the complete HTTP request. Rust type checking runs after the response arrives. The matched comparison divides total output tokens by total request time for each engine.
Repeat matched runs and publish the repetitions, spread and aggregation method. Preserve the order of runs and the conditions that could have affected them. Multipliers must name the baseline and use matched work, sampling and cache conditions. Compare ordinary engine execution separately from changes to weight layout or prompt reuse.
Run conditions
Separate cold and warm starts, reused and fresh context, and model preparation from execution. Record memory pressure, swap, power and thermal state alongside the OS, compiler, hardware and effective settings.
Measure whole-machine memory separately from process or buffer counters. Declare exclusions before the run and retain rejected runs with their reasons.
Reproduction
A reproducible report needs source and build revisions, model and tokenizer identities, template settings, test commands, input provenance and aggregate data. Private prompts and transcripts stay private.
See the recorded results.