diff --git a/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md b/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md new file mode 100644 index 0000000..8e1a00b --- /dev/null +++ b/docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md @@ -0,0 +1,885 @@ +# lq_token_test Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond. + +**Architecture:** Keep the project as one binary crate with focused internal modules: `cli`, `config`, `runner`, `protocols`, `benchmarks`, and `metrics`. Datasets are not committed; the CLI downloads them into `data/benchmarks/`, writes metadata, and benchmark commands read only local files. + +**Tech Stack:** Rust 2024, `clap`, `serde`, `serde_yaml`, `serde_json`, `tokio`, `reqwest` with Rustls, `anyhow`, `thiserror`, `tracing`, `indicatif`, `hdrhistogram`, `regex`, `sha2`, `chrono`, `futures`. + +--- + +## File Structure + +- Modify `Cargo.toml`: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies. +- Modify `.gitignore`: keep `target/` and add local benchmark data/output directories. +- Create `config.example.yaml`: example relay and benchmark configuration. +- Modify `src/main.rs`: async entry point, logging, command dispatch. +- Create `src/cli.rs`: `clap` command definitions. +- Create `src/config.rs`: YAML loading, provider lookup, environment expansion. +- Create `src/runner.rs`: provider-neutral request and response types plus request execution timing. +- Create `src/protocols/mod.rs`: protocol enum and dispatch. +- Create `src/protocols/openai.rs`: OpenAI `/chat/completions` request/response adapter. +- Create `src/protocols/anthropic.rs`: Anthropic `/v1/messages` request/response adapter. +- Create `src/benchmarks/mod.rs`: benchmark command orchestration and shared record types. +- Create `src/benchmarks/aime.rs`: AIME 2026 download, parse, prompt, judge integration. +- Create `src/benchmarks/gpqa.rs`: GPQA-Diamond download, parse, prompt, judge integration. +- Create `src/benchmarks/judge.rs`: numeric and multiple-choice answer extraction. +- Create `src/metrics.rs`: latency, success/error, and accuracy summaries. + +## Task 1: Dependencies And Project Skeleton + +**Files:** +- Modify: `Cargo.toml` +- Modify: `.gitignore` +- Create: `config.example.yaml` +- Modify: `src/main.rs` +- Create: `src/cli.rs` +- Create: `src/config.rs` +- Create: `src/runner.rs` +- Create: `src/protocols/mod.rs` +- Create: `src/protocols/openai.rs` +- Create: `src/protocols/anthropic.rs` +- Create: `src/benchmarks/mod.rs` +- Create: `src/benchmarks/aime.rs` +- Create: `src/benchmarks/gpqa.rs` +- Create: `src/benchmarks/judge.rs` +- Create: `src/metrics.rs` + +- [ ] **Step 1: Add crate dependencies** + +Set `Cargo.toml` dependencies to: + +```toml +[package] +name = "lq_token_test" +version = "0.1.0" +edition = "2024" + +[dependencies] +anyhow = "1" +chrono = { version = "0.4", features = ["serde"] } +clap = { version = "4", features = ["derive"] } +futures = "0.3" +hdrhistogram = "7" +indicatif = "0.17" +regex = "1" +reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +serde = { version = "1", features = ["derive"] } +serde_json = "1" +serde_yaml = "0.9" +sha2 = "0.10" +thiserror = "2" +tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] } +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["env-filter"] } +url = "2" + +[dev-dependencies] +tempfile = "3" +wiremock = "0.6" +``` + +- [ ] **Step 2: Ignore local generated data** + +Set `.gitignore` to: + +```gitignore +/target +/data/benchmarks +/reports +``` + +- [ ] **Step 3: Add example config** + +Create `config.example.yaml`: + +```yaml +default_provider: openai + +providers: + openai: + protocol: openai + base_url: "https://relay.example.com/v1" + api_token: "${OPENAI_RELAY_TOKEN}" + default_model: "gpt-4o-mini" + + anthropic: + protocol: anthropic + base_url: "https://relay.example.com" + api_token: "${ANTHROPIC_RELAY_TOKEN}" + default_model: "claude-3-5-sonnet-latest" + +benchmarks: + data_dir: "data/benchmarks" + aime2026: + source: "huggingface:MathArena/aime_2026" + split: "train" + gpqa_diamond: + source: "huggingface:Idavidrein/gpqa" + split: "gpqa_diamond" +``` + +- [ ] **Step 4: Create empty module declarations** + +Replace `src/main.rs` with: + +```rust +mod benchmarks; +mod cli; +mod config; +mod metrics; +mod protocols; +mod runner; + +use anyhow::Result; +use clap::Parser; +use cli::Cli; + +#[tokio::main] +async fn main() -> Result<()> { + tracing_subscriber::fmt() + .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) + .init(); + + let cli = Cli::parse(); + cli::dispatch(cli).await +} +``` + +Create placeholder modules that compile: + +```rust +// src/cli.rs +use anyhow::Result; +use clap::{Parser, Subcommand}; +use std::path::PathBuf; + +#[derive(Debug, Parser)] +#[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")] +pub struct Cli { + #[command(subcommand)] + pub command: Command, +} + +#[derive(Debug, Subcommand)] +pub enum Command { + Check { + #[arg(long, default_value = "config.yaml")] + config: PathBuf, + #[arg(long)] + provider: Option, + #[arg(long)] + model: Option, + #[arg(long)] + prompt: String, + }, + Dataset { + #[command(subcommand)] + command: DatasetCommand, + }, + Bench { + #[command(subcommand)] + command: BenchCommand, + }, + Rpm { + #[arg(long, default_value = "config.yaml")] + config: PathBuf, + #[arg(long)] + provider: Option, + #[arg(long)] + model: Option, + #[arg(long)] + rpm: u32, + #[arg(long)] + duration: String, + #[arg(long)] + prompt: String, + }, +} + +#[derive(Debug, Subcommand)] +pub enum DatasetCommand { + Fetch { dataset: String }, +} + +#[derive(Debug, Subcommand)] +pub enum BenchCommand { + Aime2026 { + #[arg(long, default_value = "config.yaml")] + config: PathBuf, + #[arg(long)] + provider: Option, + #[arg(long)] + model: Option, + #[arg(long, default_value_t = 4)] + concurrency: usize, + #[arg(long)] + limit: Option, + }, + GpqaDiamond { + #[arg(long, default_value = "config.yaml")] + config: PathBuf, + #[arg(long)] + provider: Option, + #[arg(long)] + model: Option, + #[arg(long, default_value_t = 4)] + concurrency: usize, + #[arg(long)] + limit: Option, + }, +} + +pub async fn dispatch(cli: Cli) -> Result<()> { + match cli.command { + Command::Check { .. } => anyhow::bail!("check is not implemented yet"), + Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"), + Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"), + Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"), + } +} +``` + +Create each other module with one line: + +```rust +// src/config.rs, src/runner.rs, src/metrics.rs +``` + +For directories: + +```rust +// src/protocols/mod.rs +pub mod anthropic; +pub mod openai; +``` + +```rust +// src/protocols/openai.rs +``` + +```rust +// src/protocols/anthropic.rs +``` + +```rust +// src/benchmarks/mod.rs +pub mod aime; +pub mod gpqa; +pub mod judge; +``` + +```rust +// src/benchmarks/aime.rs +``` + +```rust +// src/benchmarks/gpqa.rs +``` + +```rust +// src/benchmarks/judge.rs +``` + +- [ ] **Step 5: Verify skeleton** + +Run: `cargo fmt` + +Run: `cargo test` + +Expected: build succeeds with no tests or only placeholder warnings. + +- [ ] **Step 6: Commit** + +```bash +git add Cargo.toml Cargo.lock .gitignore config.example.yaml src +git commit -m "chore: initialize cli project skeleton" +``` + +## Task 2: Config Loading + +**Files:** +- Modify: `src/config.rs` +- Modify: `src/cli.rs` + +- [ ] **Step 1: Write config tests** + +Add tests to `src/config.rs`: + +```rust +#[cfg(test)] +mod tests { + use super::*; + use std::fs; + + #[test] + fn loads_provider_and_expands_env_token() { + unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") }; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("config.yaml"); + fs::write( + &path, + r#" +default_provider: openai +providers: + openai: + protocol: openai + base_url: "https://example.com/v1" + api_token: "${LQ_TEST_TOKEN}" + default_model: "gpt-test" +benchmarks: + data_dir: "data/benchmarks" +"#, + ) + .unwrap(); + + let config = AppConfig::load(&path).unwrap(); + let provider = config.provider(None).unwrap(); + assert_eq!(provider.api_token, "secret-token"); + assert_eq!(provider.default_model, "gpt-test"); + assert_eq!(provider.protocol, ProtocolKind::Openai); + } + + #[test] + fn rejects_missing_env_token() { + unsafe { std::env::remove_var("LQ_MISSING_TOKEN") }; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("config.yaml"); + fs::write( + &path, + r#" +default_provider: openai +providers: + openai: + protocol: openai + base_url: "https://example.com/v1" + api_token: "${LQ_MISSING_TOKEN}" + default_model: "gpt-test" +"#, + ) + .unwrap(); + + let err = AppConfig::load(&path).unwrap_err().to_string(); + assert!(err.contains("LQ_MISSING_TOKEN")); + } +} +``` + +- [ ] **Step 2: Run tests to verify failure** + +Run: `cargo test config::tests` + +Expected: fails because `AppConfig` and `ProtocolKind` are not implemented. + +- [ ] **Step 3: Implement config** + +Implement `src/config.rs`: + +```rust +use serde::Deserialize; +use std::{collections::BTreeMap, fs, path::Path}; +use thiserror::Error; + +#[derive(Debug, Error)] +pub enum ConfigError { + #[error("failed to read config {path}: {source}")] + Read { + path: String, + #[source] + source: std::io::Error, + }, + #[error("failed to parse yaml config: {0}")] + Parse(#[from] serde_yaml::Error), + #[error("unknown provider '{0}'")] + UnknownProvider(String), + #[error("config has no default_provider and no --provider was supplied")] + MissingProvider, + #[error("environment variable '{0}' referenced by config is not set")] + MissingEnv(String), +} + +#[derive(Debug, Clone, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ProtocolKind { + Openai, + Anthropic, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct ProviderConfig { + pub protocol: ProtocolKind, + pub base_url: String, + pub api_token: String, + pub default_model: String, +} + +#[derive(Debug, Clone, Deserialize, Default)] +pub struct BenchmarkConfig { + #[serde(default = "default_data_dir")] + pub data_dir: String, + #[serde(default)] + pub aime2026: Option, + #[serde(default)] + pub gpqa_diamond: Option, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct DatasetConfig { + pub source: String, + pub split: String, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct AppConfig { + pub default_provider: Option, + pub providers: BTreeMap, + #[serde(default)] + pub benchmarks: BenchmarkConfig, +} + +fn default_data_dir() -> String { + "data/benchmarks".to_string() +} + +impl AppConfig { + pub fn load(path: &Path) -> Result { + let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read { + path: path.display().to_string(), + source, + })?; + let expanded = expand_env_refs(&raw)?; + Ok(serde_yaml::from_str(&expanded)?) + } + + pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> { + let name = match provider { + Some(name) => name, + None => self + .default_provider + .as_deref() + .ok_or(ConfigError::MissingProvider)?, + }; + self.providers + .get(name) + .ok_or_else(|| ConfigError::UnknownProvider(name.to_string())) + } +} + +fn expand_env_refs(input: &str) -> Result { + let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex"); + let mut output = String::with_capacity(input.len()); + let mut last = 0; + + for caps in re.captures_iter(input) { + let whole = caps.get(0).expect("whole match"); + let name = caps.get(1).expect("env name").as_str(); + output.push_str(&input[last..whole.start()]); + let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?; + output.push_str(&value); + last = whole.end(); + } + + output.push_str(&input[last..]); + Ok(output) +} +``` + +- [ ] **Step 4: Run tests** + +Run: `cargo test config::tests` + +Expected: both config tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add src/config.rs src/cli.rs +git commit -m "feat: load yaml relay config" +``` + +## Task 3: Dataset Download And Parsing + +**Files:** +- Modify: `src/cli.rs` +- Modify: `src/benchmarks/mod.rs` +- Modify: `src/benchmarks/aime.rs` +- Modify: `src/benchmarks/gpqa.rs` +- Modify: `src/benchmarks/judge.rs` + +- [ ] **Step 1: Add judge tests** + +Add tests to `src/benchmarks/judge.rs`: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn extracts_final_integer() { + assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string())); + assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string())); + } + + #[test] + fn extracts_choice_letter() { + assert_eq!(extract_choice("Answer: C"), Some('C')); + assert_eq!(extract_choice("I choose (b)."), Some('B')); + } +} +``` + +- [ ] **Step 2: Implement judge functions** + +Implement `src/benchmarks/judge.rs`: + +```rust +pub fn extract_final_integer(text: &str) -> Option { + let re = regex::Regex::new(r"-?\d+").expect("valid integer regex"); + re.find_iter(text).last().map(|m| m.as_str().to_string()) +} + +pub fn extract_choice(text: &str) -> Option { + let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex"); + re.captures_iter(text) + .last() + .and_then(|caps| caps.get(1)) + .and_then(|m| m.as_str().chars().next()) + .map(|c| c.to_ascii_uppercase()) +} + +pub fn judge_integer(output: &str, expected: &str) -> bool { + extract_final_integer(output).as_deref() == Some(expected.trim()) +} + +pub fn judge_choice(output: &str, expected: char) -> bool { + extract_choice(output) == Some(expected.to_ascii_uppercase()) +} +``` + +- [ ] **Step 3: Add dataset parse tests with small fixtures** + +Add tests in `src/benchmarks/aime.rs` and `src/benchmarks/gpqa.rs` using inline rows: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn builds_aime_prompt() { + let case = AimeCase { + id: "1".to_string(), + problem: "What is 20 + 22?".to_string(), + answer: "42".to_string(), + }; + assert!(case.prompt().contains("What is 20 + 22?")); + assert!(case.prompt().contains("final integer answer")); + } +} +``` + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn builds_gpqa_prompt() { + let case = GpqaCase { + id: "gpqa_1".to_string(), + question: "Which option is correct?".to_string(), + choices: [ + ("A".to_string(), "Alpha".to_string()), + ("B".to_string(), "Beta".to_string()), + ("C".to_string(), "Gamma".to_string()), + ("D".to_string(), "Delta".to_string()), + ], + answer: 'B', + }; + let prompt = case.prompt(); + assert!(prompt.contains("A. Alpha")); + assert!(prompt.contains("answer with exactly one letter")); + } +} +``` + +- [ ] **Step 4: Implement benchmark case types and fetch stubs** + +Implement enough for prompts and later download wiring: + +```rust +// src/benchmarks/aime.rs +#[derive(Debug, Clone)] +pub struct AimeCase { + pub id: String, + pub problem: String, + pub answer: String, +} + +impl AimeCase { + pub fn prompt(&self) -> String { + format!( + "Solve the following AIME problem. Return only the final integer answer.\n\n{}", + self.problem + ) + } +} +``` + +```rust +// src/benchmarks/gpqa.rs +#[derive(Debug, Clone)] +pub struct GpqaCase { + pub id: String, + pub question: String, + pub choices: [(String, String); 4], + pub answer: char, +} + +impl GpqaCase { + pub fn prompt(&self) -> String { + format!( + "Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}", + self.question, + self.choices[0].0, + self.choices[0].1, + self.choices[1].0, + self.choices[1].1, + self.choices[2].0, + self.choices[2].1, + self.choices[3].0, + self.choices[3].1 + ) + } +} +``` + +- [ ] **Step 5: Implement `dataset fetch`** + +Add download functions that use Hugging Face raw URLs: + +```rust +// src/benchmarks/mod.rs +use anyhow::{Context, Result}; +use chrono::Utc; +use sha2::{Digest, Sha256}; +use std::{fs, path::{Path, PathBuf}}; + +pub mod aime; +pub mod gpqa; +pub mod judge; + +pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result { + match dataset { + "aime2026" => fetch_to_dir( + "aime2026", + "https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet", + data_dir, + None, + ) + .await, + "gpqa-diamond" => fetch_to_dir( + "gpqa_diamond", + "https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv", + data_dir, + std::env::var("HF_TOKEN").ok(), + ) + .await, + other => anyhow::bail!("unknown dataset '{other}'"), + } +} + +async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option) -> Result { + let dir = data_dir.join(name); + fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?; + let filename = url.rsplit('/').next().unwrap_or("dataset"); + let output = dir.join(filename); + + let client = reqwest::Client::new(); + let mut request = client.get(url); + if let Some(token) = token { + request = request.bearer_auth(token); + } + let bytes = request.send().await?.error_for_status()?.bytes().await?; + fs::write(&output, &bytes)?; + + let hash = Sha256::digest(&bytes); + let metadata = format!( + "name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n", + Utc::now().to_rfc3339(), + hash, + bytes.len() + ); + fs::write(dir.join("metadata.yaml"), metadata)?; + Ok(output) +} +``` + +- [ ] **Step 6: Wire dataset command** + +Update `src/cli.rs` dispatch for `DatasetCommand::Fetch` to load default data dir from config when `config.yaml` exists, otherwise use `data/benchmarks`. + +- [ ] **Step 7: Run tests** + +Run: `cargo test benchmarks` + +Expected: judge and prompt tests pass. Network fetch is not tested by default. + +- [ ] **Step 8: Commit** + +```bash +git add src/benchmarks src/cli.rs +git commit -m "feat: add benchmark dataset fetch and prompts" +``` + +## Task 4: Protocol Adapters And Single Check + +**Files:** +- Modify: `src/runner.rs` +- Modify: `src/protocols/mod.rs` +- Modify: `src/protocols/openai.rs` +- Modify: `src/protocols/anthropic.rs` +- Modify: `src/cli.rs` + +- [ ] **Step 1: Write HTTP adapter tests with wiremock** + +Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text. + +- [ ] **Step 2: Define runner types** + +Implement provider-neutral types: + +```rust +#[derive(Debug, Clone)] +pub struct ModelRequest { + pub base_url: String, + pub api_token: String, + pub model: String, + pub prompt: String, + pub temperature: f32, + pub max_tokens: u32, +} + +#[derive(Debug, Clone)] +pub struct ModelResponse { + pub text: String, + pub status: u16, + pub elapsed_ms: u128, +} +``` + +- [ ] **Step 3: Implement OpenAI adapter** + +Post to `{base_url}/chat/completions` with `Authorization: Bearer ` and parse `choices[0].message.content`. + +- [ ] **Step 4: Implement Anthropic adapter** + +Post to `{base_url}/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and parse the first text block in `content`. + +- [ ] **Step 5: Wire `check` command** + +Load config, resolve provider and model, run one request, then print status, elapsed time, and response text. + +- [ ] **Step 6: Run tests** + +Run: `cargo test protocols runner` + +Expected: mocked protocol tests pass. + +- [ ] **Step 7: Commit** + +```bash +git add src/runner.rs src/protocols src/cli.rs +git commit -m "feat: add openai and anthropic request adapters" +``` + +## Task 5: Metrics, Benchmark Runs, And RPM + +**Files:** +- Modify: `src/metrics.rs` +- Modify: `src/benchmarks/mod.rs` +- Modify: `src/benchmarks/aime.rs` +- Modify: `src/benchmarks/gpqa.rs` +- Modify: `src/cli.rs` + +- [ ] **Step 1: Add metrics tests** + +Test success/error counts, accuracy, and latency percentile calculation. + +- [ ] **Step 2: Implement metrics summary** + +Use `hdrhistogram::Histogram` for elapsed milliseconds and counters for success, failure, correct, and total judged. + +- [ ] **Step 3: Implement local benchmark loading** + +For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch `. + +- [ ] **Step 4: Implement benchmark execution** + +Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics. + +- [ ] **Step 5: Implement RPM command** + +Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary. + +- [ ] **Step 6: Run tests** + +Run: `cargo test` + +Expected: unit tests pass. + +- [ ] **Step 7: Commit** + +```bash +git add src/metrics.rs src/benchmarks src/cli.rs +git commit -m "feat: run benchmark and rpm tests" +``` + +## Task 6: Verification And Docs + +**Files:** +- Create: `README.md` +- Modify: `config.example.yaml` + +- [ ] **Step 1: Document usage** + +Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test. + +- [ ] **Step 2: Verify formatting and tests** + +Run: + +```bash +cargo fmt --check +cargo test +cargo clippy --all-targets -- -D warnings +``` + +Expected: all pass. + +- [ ] **Step 3: Manual smoke commands** + +Run: + +```bash +cargo run -- --help +cargo run -- dataset fetch aime2026 +``` + +Expected: help renders; AIME fetch creates `data/benchmarks/aime2026/metadata.yaml`. + +- [ ] **Step 4: Commit** + +```bash +git add README.md config.example.yaml +git commit -m "docs: add usage guide" +``` + +## Self-Review + +- Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage. +- Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior. +- Type consistency: dataset names are `aime2026` and `gpqa-diamond` in CLI; local directory names are `aime2026` and `gpqa_diamond`; module names are `aime` and `gpqa`.