diff --git a/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md b/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md index 8e1a00b..ce0cd4b 100644 --- a/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md +++ b/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md @@ -804,6 +804,7 @@ git commit -m "feat: add openai and anthropic request adapters" - Modify: `src/benchmarks/aime.rs` - Modify: `src/benchmarks/gpqa.rs` - Modify: `src/cli.rs` +- Create: `src/report.rs` - [ ] **Step 1: Add metrics tests** @@ -813,6 +814,89 @@ Test success/error counts, accuracy, and latency percentile calculation. Use `hdrhistogram::Histogram` for elapsed milliseconds and counters for success, failure, correct, and total judged. +- [ ] **Step 2.5: Implement terminal and JSON reports** + +Create `src/report.rs` with serializable report structs and a writer that creates `reports/` when needed. Benchmark runs must print a terminal summary and also save a JSON report. + +Terminal output for benchmark runs must include: + +```text +accuracy: 83.33% +success: 30/30 +latency: + p50: 1240 ms + p95: 2860 ms + p99: 3120 ms +errors: + http_429: 2 +wrong cases: + - id: aime2026_007 + expected: 42 + actual: 41 +report: reports/aime2026-openai-gpt-4o-mini-20260506-153012.json +``` + +Benchmark report JSON must include: + +```json +{ + "benchmark": "aime2026", + "provider": "openai", + "model": "gpt-4o-mini", + "dataset": { + "source": "huggingface:MathArena/aime_2026", + "split": "train", + "revision": null, + "local_path": "data/benchmarks/aime2026/train-00000-of-00001.parquet" + }, + "run": { + "started_at": "2026-05-06T15:30:12Z", + "duration_ms": 48210, + "concurrency": 4, + "limit": 30, + "temperature": 0.0, + "max_tokens": 1024 + }, + "summary": { + "accuracy": 0.8333, + "success": 30, + "total": 30, + "correct": 25, + "wrong": 5, + "failed": 0, + "latency_ms": { + "p50": 1240, + "p95": 2860, + "p99": 3120 + } + }, + "errors": [ + { + "code": "http_429", + "count": 2 + } + ], + "wrong_cases": [ + { + "id": "aime2026_007", + "question": "problem text", + "expected": "42", + "actual": "41", + "raw_output": "41" + } + ] +} +``` + +Default report behavior: + +- Save complete `wrong_cases`. +- Do not save every successful case. +- Use a readable filename shape: `---.json`. +- Sanitize model names for filenames. +- Print the report path after writing the file. +- RPM reports should use the same writer style, but omit accuracy and wrong cases. + - [ ] **Step 3: Implement local benchmark loading** For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch `. @@ -821,10 +905,14 @@ For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If dat Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics. +After benchmark execution finishes, populate the report model and write the JSON report even when some cases fail. + - [ ] **Step 5: Implement RPM command** Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary. +RPM terminal output and JSON report must include target RPM, actual request count, success count, failure count, latency p50/p95/p99, and errors by status/code. + - [ ] **Step 6: Run tests** Run: `cargo test` @@ -834,7 +922,7 @@ Expected: unit tests pass. - [ ] **Step 7: Commit** ```bash -git add src/metrics.rs src/benchmarks src/cli.rs +git add src/metrics.rs src/report.rs src/benchmarks src/cli.rs git commit -m "feat: run benchmark and rpm tests" ```