# lq_token_test Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond.

**Architecture:** Keep the project as one binary crate with focused internal modules: `cli`, `config`, `runner`, `protocols`, `benchmarks`, and `metrics`. Datasets are not committed; the CLI downloads them into `data/benchmarks/`, writes metadata, and benchmark commands read only local files.

**Tech Stack:** Rust 2024, `clap`, `serde`, `serde_yaml`, `serde_json`, `tokio`, `reqwest` with Rustls, `anyhow`, `thiserror`, `tracing`, `indicatif`, `hdrhistogram`, `regex`, `sha2`, `chrono`, `futures`.

---

## File Structure

- Modify `Cargo.toml`: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies.
- Modify `.gitignore`: keep `target/` and add local benchmark data/output directories.
- Create `config.example.yaml`: example relay and benchmark configuration.
- Modify `src/main.rs`: async entry point, logging, command dispatch.
- Create `src/cli.rs`: `clap` command definitions.
- Create `src/config.rs`: YAML loading, provider lookup, environment expansion.
- Create `src/runner.rs`: provider-neutral request and response types plus request execution timing.
- Create `src/protocols/mod.rs`: protocol enum and dispatch.
- Create `src/protocols/openai.rs`: OpenAI `/chat/completions` request/response adapter.
- Create `src/protocols/anthropic.rs`: Anthropic `/v1/messages` request/response adapter.
- Create `src/benchmarks/mod.rs`: benchmark command orchestration and shared record types.
- Create `src/benchmarks/aime.rs`: AIME 2026 download, parse, prompt, judge integration.
- Create `src/benchmarks/gpqa.rs`: GPQA-Diamond download, parse, prompt, judge integration.
- Create `src/benchmarks/judge.rs`: numeric and multiple-choice answer extraction.
- Create `src/metrics.rs`: latency, success/error, and accuracy summaries.

## Task 1: Dependencies And Project Skeleton

**Files:**
- Modify: `Cargo.toml`
- Modify: `.gitignore`
- Create: `config.example.yaml`
- Modify: `src/main.rs`
- Create: `src/cli.rs`
- Create: `src/config.rs`
- Create: `src/runner.rs`
- Create: `src/protocols/mod.rs`
- Create: `src/protocols/openai.rs`
- Create: `src/protocols/anthropic.rs`
- Create: `src/benchmarks/mod.rs`
- Create: `src/benchmarks/aime.rs`
- Create: `src/benchmarks/gpqa.rs`
- Create: `src/benchmarks/judge.rs`
- Create: `src/metrics.rs`

- [ ] **Step 1: Add crate dependencies**

Set `Cargo.toml` dependencies to:

```toml
[package]
name = "lq_token_test"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "1"
chrono = { version = "0.4", features = ["serde"] }
clap = { version = "4", features = ["derive"] }
futures = "0.3"
hdrhistogram = "7"
indicatif = "0.17"
regex = "1"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
sha2 = "0.10"
thiserror = "2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
url = "2"

[dev-dependencies]
tempfile = "3"
wiremock = "0.6"
```

- [ ] **Step 2: Ignore local generated data**

Set `.gitignore` to:

```gitignore
/target
/data/benchmarks
/reports
```

- [ ] **Step 3: Add example config**

Create `config.example.yaml`:

```yaml
default_provider: openai

providers:
  openai:
    protocol: openai
    base_url: "https://relay.example.com/v1"
    api_token: "${OPENAI_RELAY_TOKEN}"
    default_model: "gpt-4o-mini"

  anthropic:
    protocol: anthropic
    base_url: "https://relay.example.com"
    api_token: "${ANTHROPIC_RELAY_TOKEN}"
    default_model: "claude-3-5-sonnet-latest"

benchmarks:
  data_dir: "data/benchmarks"
  aime2026:
    source: "huggingface:MathArena/aime_2026"
    split: "train"
  gpqa_diamond:
    source: "huggingface:Idavidrein/gpqa"
    split: "gpqa_diamond"
```

- [ ] **Step 4: Create empty module declarations**

Replace `src/main.rs` with:

```rust
mod benchmarks;
mod cli;
mod config;
mod metrics;
mod protocols;
mod runner;

use anyhow::Result;
use clap::Parser;
use cli::Cli;

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
        .init();

    let cli = Cli::parse();
    cli::dispatch(cli).await
}
```

Create placeholder modules that compile:

```rust
// src/cli.rs
use anyhow::Result;
use clap::{Parser, Subcommand};
use std::path::PathBuf;

#[derive(Debug, Parser)]
#[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")]
pub struct Cli {
    #[command(subcommand)]
    pub command: Command,
}

#[derive(Debug, Subcommand)]
pub enum Command {
    Check {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long)]
        prompt: String,
    },
    Dataset {
        #[command(subcommand)]
        command: DatasetCommand,
    },
    Bench {
        #[command(subcommand)]
        command: BenchCommand,
    },
    Rpm {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long)]
        rpm: u32,
        #[arg(long)]
        duration: String,
        #[arg(long)]
        prompt: String,
    },
}

#[derive(Debug, Subcommand)]
pub enum DatasetCommand {
    Fetch { dataset: String },
}

#[derive(Debug, Subcommand)]
pub enum BenchCommand {
    Aime2026 {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long, default_value_t = 4)]
        concurrency: usize,
        #[arg(long)]
        limit: Option<usize>,
    },
    GpqaDiamond {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long, default_value_t = 4)]
        concurrency: usize,
        #[arg(long)]
        limit: Option<usize>,
    },
}

pub async fn dispatch(cli: Cli) -> Result<()> {
    match cli.command {
        Command::Check { .. } => anyhow::bail!("check is not implemented yet"),
        Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"),
        Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"),
        Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"),
    }
}
```

Create each other module with one line:

```rust
// src/config.rs, src/runner.rs, src/metrics.rs
```

For directories:

```rust
// src/protocols/mod.rs
pub mod anthropic;
pub mod openai;
```

```rust
// src/protocols/openai.rs
```

```rust
// src/protocols/anthropic.rs
```

```rust
// src/benchmarks/mod.rs
pub mod aime;
pub mod gpqa;
pub mod judge;
```

```rust
// src/benchmarks/aime.rs
```

```rust
// src/benchmarks/gpqa.rs
```

```rust
// src/benchmarks/judge.rs
```

- [ ] **Step 5: Verify skeleton**

Run: `cargo fmt`

Run: `cargo test`

Expected: build succeeds with no tests or only placeholder warnings.

- [ ] **Step 6: Commit**

```bash
git add Cargo.toml Cargo.lock .gitignore config.example.yaml src
git commit -m "chore: initialize cli project skeleton"
```

## Task 2: Config Loading

**Files:**
- Modify: `src/config.rs`
- Modify: `src/cli.rs`

- [ ] **Step 1: Write config tests**

Add tests to `src/config.rs`:

```rust
#[cfg(test)]
mod tests {
    use super::*;
    use std::fs;

    #[test]
    fn loads_provider_and_expands_env_token() {
        unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") };
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("config.yaml");
        fs::write(
            &path,
            r#"
default_provider: openai
providers:
  openai:
    protocol: openai
    base_url: "https://example.com/v1"
    api_token: "${LQ_TEST_TOKEN}"
    default_model: "gpt-test"
benchmarks:
  data_dir: "data/benchmarks"
"#,
        )
        .unwrap();

        let config = AppConfig::load(&path).unwrap();
        let provider = config.provider(None).unwrap();
        assert_eq!(provider.api_token, "secret-token");
        assert_eq!(provider.default_model, "gpt-test");
        assert_eq!(provider.protocol, ProtocolKind::Openai);
    }

    #[test]
    fn rejects_missing_env_token() {
        unsafe { std::env::remove_var("LQ_MISSING_TOKEN") };
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("config.yaml");
        fs::write(
            &path,
            r#"
default_provider: openai
providers:
  openai:
    protocol: openai
    base_url: "https://example.com/v1"
    api_token: "${LQ_MISSING_TOKEN}"
    default_model: "gpt-test"
"#,
        )
        .unwrap();

        let err = AppConfig::load(&path).unwrap_err().to_string();
        assert!(err.contains("LQ_MISSING_TOKEN"));
    }
}
```

- [ ] **Step 2: Run tests to verify failure**

Run: `cargo test config::tests`

Expected: fails because `AppConfig` and `ProtocolKind` are not implemented.

- [ ] **Step 3: Implement config**

Implement `src/config.rs`:

```rust
use serde::Deserialize;
use std::{collections::BTreeMap, fs, path::Path};
use thiserror::Error;

#[derive(Debug, Error)]
pub enum ConfigError {
    #[error("failed to read config {path}: {source}")]
    Read {
        path: String,
        #[source]
        source: std::io::Error,
    },
    #[error("failed to parse yaml config: {0}")]
    Parse(#[from] serde_yaml::Error),
    #[error("unknown provider '{0}'")]
    UnknownProvider(String),
    #[error("config has no default_provider and no --provider was supplied")]
    MissingProvider,
    #[error("environment variable '{0}' referenced by config is not set")]
    MissingEnv(String),
}

#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum ProtocolKind {
    Openai,
    Anthropic,
}

#[derive(Debug, Clone, Deserialize)]
pub struct ProviderConfig {
    pub protocol: ProtocolKind,
    pub base_url: String,
    pub api_token: String,
    pub default_model: String,
}

#[derive(Debug, Clone, Deserialize, Default)]
pub struct BenchmarkConfig {
    #[serde(default = "default_data_dir")]
    pub data_dir: String,
    #[serde(default)]
    pub aime2026: Option<DatasetConfig>,
    #[serde(default)]
    pub gpqa_diamond: Option<DatasetConfig>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct DatasetConfig {
    pub source: String,
    pub split: String,
}

#[derive(Debug, Clone, Deserialize)]
pub struct AppConfig {
    pub default_provider: Option<String>,
    pub providers: BTreeMap<String, ProviderConfig>,
    #[serde(default)]
    pub benchmarks: BenchmarkConfig,
}

fn default_data_dir() -> String {
    "data/benchmarks".to_string()
}

impl AppConfig {
    pub fn load(path: &Path) -> Result<Self, ConfigError> {
        let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read {
            path: path.display().to_string(),
            source,
        })?;
        let expanded = expand_env_refs(&raw)?;
        Ok(serde_yaml::from_str(&expanded)?)
    }

    pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> {
        let name = match provider {
            Some(name) => name,
            None => self
                .default_provider
                .as_deref()
                .ok_or(ConfigError::MissingProvider)?,
        };
        self.providers
            .get(name)
            .ok_or_else(|| ConfigError::UnknownProvider(name.to_string()))
    }
}

fn expand_env_refs(input: &str) -> Result<String, ConfigError> {
    let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex");
    let mut output = String::with_capacity(input.len());
    let mut last = 0;

    for caps in re.captures_iter(input) {
        let whole = caps.get(0).expect("whole match");
        let name = caps.get(1).expect("env name").as_str();
        output.push_str(&input[last..whole.start()]);
        let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?;
        output.push_str(&value);
        last = whole.end();
    }

    output.push_str(&input[last..]);
    Ok(output)
}
```

- [ ] **Step 4: Run tests**

Run: `cargo test config::tests`

Expected: both config tests pass.

- [ ] **Step 5: Commit**

```bash
git add src/config.rs src/cli.rs
git commit -m "feat: load yaml relay config"
```

## Task 3: Dataset Download And Parsing

**Files:**
- Modify: `src/cli.rs`
- Modify: `src/benchmarks/mod.rs`
- Modify: `src/benchmarks/aime.rs`
- Modify: `src/benchmarks/gpqa.rs`
- Modify: `src/benchmarks/judge.rs`

- [ ] **Step 1: Add judge tests**

Add tests to `src/benchmarks/judge.rs`:

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn extracts_final_integer() {
        assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string()));
        assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string()));
    }

    #[test]
    fn extracts_choice_letter() {
        assert_eq!(extract_choice("Answer: C"), Some('C'));
        assert_eq!(extract_choice("I choose (b)."), Some('B'));
    }
}
```

- [ ] **Step 2: Implement judge functions**

Implement `src/benchmarks/judge.rs`:

```rust
pub fn extract_final_integer(text: &str) -> Option<String> {
    let re = regex::Regex::new(r"-?\d+").expect("valid integer regex");
    re.find_iter(text).last().map(|m| m.as_str().to_string())
}

pub fn extract_choice(text: &str) -> Option<char> {
    let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex");
    re.captures_iter(text)
        .last()
        .and_then(|caps| caps.get(1))
        .and_then(|m| m.as_str().chars().next())
        .map(|c| c.to_ascii_uppercase())
}

pub fn judge_integer(output: &str, expected: &str) -> bool {
    extract_final_integer(output).as_deref() == Some(expected.trim())
}

pub fn judge_choice(output: &str, expected: char) -> bool {
    extract_choice(output) == Some(expected.to_ascii_uppercase())
}
```

- [ ] **Step 3: Add dataset parse tests with small fixtures**

Add tests in `src/benchmarks/aime.rs` and `src/benchmarks/gpqa.rs` using inline rows:

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn builds_aime_prompt() {
        let case = AimeCase {
            id: "1".to_string(),
            problem: "What is 20 + 22?".to_string(),
            answer: "42".to_string(),
        };
        assert!(case.prompt().contains("What is 20 + 22?"));
        assert!(case.prompt().contains("final integer answer"));
    }
}
```

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn builds_gpqa_prompt() {
        let case = GpqaCase {
            id: "gpqa_1".to_string(),
            question: "Which option is correct?".to_string(),
            choices: [
                ("A".to_string(), "Alpha".to_string()),
                ("B".to_string(), "Beta".to_string()),
                ("C".to_string(), "Gamma".to_string()),
                ("D".to_string(), "Delta".to_string()),
            ],
            answer: 'B',
        };
        let prompt = case.prompt();
        assert!(prompt.contains("A. Alpha"));
        assert!(prompt.contains("answer with exactly one letter"));
    }
}
```

- [ ] **Step 4: Implement benchmark case types and fetch stubs**

Implement enough for prompts and later download wiring:

```rust
// src/benchmarks/aime.rs
#[derive(Debug, Clone)]
pub struct AimeCase {
    pub id: String,
    pub problem: String,
    pub answer: String,
}

impl AimeCase {
    pub fn prompt(&self) -> String {
        format!(
            "Solve the following AIME problem. Return only the final integer answer.\n\n{}",
            self.problem
        )
    }
}
```

```rust
// src/benchmarks/gpqa.rs
#[derive(Debug, Clone)]
pub struct GpqaCase {
    pub id: String,
    pub question: String,
    pub choices: [(String, String); 4],
    pub answer: char,
}

impl GpqaCase {
    pub fn prompt(&self) -> String {
        format!(
            "Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}",
            self.question,
            self.choices[0].0,
            self.choices[0].1,
            self.choices[1].0,
            self.choices[1].1,
            self.choices[2].0,
            self.choices[2].1,
            self.choices[3].0,
            self.choices[3].1
        )
    }
}
```

- [ ] **Step 5: Implement `dataset fetch`**

Add download functions that use Hugging Face raw URLs:

```rust
// src/benchmarks/mod.rs
use anyhow::{Context, Result};
use chrono::Utc;
use sha2::{Digest, Sha256};
use std::{fs, path::{Path, PathBuf}};

pub mod aime;
pub mod gpqa;
pub mod judge;

pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> {
    match dataset {
        "aime2026" => fetch_to_dir(
            "aime2026",
            "https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet",
            data_dir,
            None,
        )
        .await,
        "gpqa-diamond" => fetch_to_dir(
            "gpqa_diamond",
            "https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv",
            data_dir,
            std::env::var("HF_TOKEN").ok(),
        )
        .await,
        other => anyhow::bail!("unknown dataset '{other}'"),
    }
}

async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> {
    let dir = data_dir.join(name);
    fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?;
    let filename = url.rsplit('/').next().unwrap_or("dataset");
    let output = dir.join(filename);

    let client = reqwest::Client::new();
    let mut request = client.get(url);
    if let Some(token) = token {
        request = request.bearer_auth(token);
    }
    let bytes = request.send().await?.error_for_status()?.bytes().await?;
    fs::write(&output, &bytes)?;

    let hash = Sha256::digest(&bytes);
    let metadata = format!(
        "name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n",
        Utc::now().to_rfc3339(),
        hash,
        bytes.len()
    );
    fs::write(dir.join("metadata.yaml"), metadata)?;
    Ok(output)
}
```

- [ ] **Step 6: Wire dataset command**

Update `src/cli.rs` dispatch for `DatasetCommand::Fetch` to load default data dir from config when `config.yaml` exists, otherwise use `data/benchmarks`.

- [ ] **Step 7: Run tests**

Run: `cargo test benchmarks`

Expected: judge and prompt tests pass. Network fetch is not tested by default.

- [ ] **Step 8: Commit**

```bash
git add src/benchmarks src/cli.rs
git commit -m "feat: add benchmark dataset fetch and prompts"
```

## Task 4: Protocol Adapters And Single Check

**Files:**
- Modify: `src/runner.rs`
- Modify: `src/protocols/mod.rs`
- Modify: `src/protocols/openai.rs`
- Modify: `src/protocols/anthropic.rs`
- Modify: `src/cli.rs`

- [ ] **Step 1: Write HTTP adapter tests with wiremock**

Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text.

- [ ] **Step 2: Define runner types**

Implement provider-neutral types:

```rust
#[derive(Debug, Clone)]
pub struct ModelRequest {
    pub base_url: String,
    pub api_token: String,
    pub model: String,
    pub prompt: String,
    pub temperature: f32,
    pub max_tokens: u32,
}

#[derive(Debug, Clone)]
pub struct ModelResponse {
    pub text: String,
    pub status: u16,
    pub elapsed_ms: u128,
}
```

- [ ] **Step 3: Implement OpenAI adapter**

Post to `{base_url}/chat/completions` with `Authorization: Bearer <token>` and parse `choices[0].message.content`.

- [ ] **Step 4: Implement Anthropic adapter**

Post to `{base_url}/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and parse the first text block in `content`.

- [ ] **Step 5: Wire `check` command**

Load config, resolve provider and model, run one request, then print status, elapsed time, and response text.

- [ ] **Step 6: Run tests**

Run: `cargo test protocols runner`

Expected: mocked protocol tests pass.

- [ ] **Step 7: Commit**

```bash
git add src/runner.rs src/protocols src/cli.rs
git commit -m "feat: add openai and anthropic request adapters"
```

## Task 5: Metrics, Benchmark Runs, And RPM

**Files:**
- Modify: `src/metrics.rs`
- Modify: `src/benchmarks/mod.rs`
- Modify: `src/benchmarks/aime.rs`
- Modify: `src/benchmarks/gpqa.rs`
- Modify: `src/cli.rs`
- Create: `src/report.rs`

- [ ] **Step 1: Add metrics tests**

Test success/error counts, accuracy, and latency percentile calculation.

- [ ] **Step 2: Implement metrics summary**

Use `hdrhistogram::Histogram<u64>` for elapsed milliseconds and counters for success, failure, correct, and total judged.

- [ ] **Step 2.5: Implement terminal and JSON reports**

Create `src/report.rs` with serializable report structs and a writer that creates `reports/` when needed. Benchmark runs must print a terminal summary and also save a JSON report.

Terminal output for benchmark runs must include:

```text
accuracy: 83.33%
success: 30/30
latency:
  p50: 1240 ms
  p95: 2860 ms
  p99: 3120 ms
errors:
  http_429: 2
wrong cases:
  - id: aime2026_007
    expected: 42
    actual: 41
report: reports/aime2026-openai-gpt-4o-mini-20260506-153012.json
```

Benchmark report JSON must include:

```json
{
  "benchmark": "aime2026",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "dataset": {
    "source": "huggingface:MathArena/aime_2026",
    "split": "train",
    "revision": null,
    "local_path": "data/benchmarks/aime2026/train-00000-of-00001.parquet"
  },
  "run": {
    "started_at": "2026-05-06T15:30:12Z",
    "duration_ms": 48210,
    "concurrency": 4,
    "limit": 30,
    "temperature": 0.0,
    "max_tokens": 1024
  },
  "summary": {
    "accuracy": 0.8333,
    "success": 30,
    "total": 30,
    "correct": 25,
    "wrong": 5,
    "failed": 0,
    "latency_ms": {
      "p50": 1240,
      "p95": 2860,
      "p99": 3120
    }
  },
  "errors": [
    {
      "code": "http_429",
      "count": 2
    }
  ],
  "wrong_cases": [
    {
      "id": "aime2026_007",
      "question": "problem text",
      "expected": "42",
      "actual": "41",
      "raw_output": "41"
    }
  ]
}
```

Default report behavior:

- Save complete `wrong_cases`.
- Do not save every successful case.
- Use a readable filename shape: `<benchmark>-<provider>-<model>-<timestamp>.json`.
- Sanitize model names for filenames.
- Print the report path after writing the file.
- RPM reports should use the same writer style, but omit accuracy and wrong cases.

- [ ] **Step 3: Implement local benchmark loading**

For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch <name>`.

- [ ] **Step 4: Implement benchmark execution**

Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics.

After benchmark execution finishes, populate the report model and write the JSON report even when some cases fail.

- [ ] **Step 5: Implement RPM command**

Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary.

RPM terminal output and JSON report must include target RPM, actual request count, success count, failure count, latency p50/p95/p99, and errors by status/code.

- [ ] **Step 6: Run tests**

Run: `cargo test`

Expected: unit tests pass.

- [ ] **Step 7: Commit**

```bash
git add src/metrics.rs src/report.rs src/benchmarks src/cli.rs
git commit -m "feat: run benchmark and rpm tests"
```

## Task 6: Verification And Docs

**Files:**
- Create: `README.md`
- Modify: `config.example.yaml`

- [ ] **Step 1: Document usage**

Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test.

- [ ] **Step 2: Verify formatting and tests**

Run:

```bash
cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
```

Expected: all pass.

- [ ] **Step 3: Manual smoke commands**

Run:

```bash
cargo run -- --help
cargo run -- dataset fetch aime2026
```

Expected: help renders; AIME fetch creates `data/benchmarks/aime2026/metadata.yaml`.

- [ ] **Step 4: Commit**

```bash
git add README.md config.example.yaml
git commit -m "docs: add usage guide"
```

## Self-Review

- Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage.
- Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior.
- Type consistency: dataset names are `aime2026` and `gpqa-diamond` in CLI; local directory names are `aime2026` and `gpqa_diamond`; module names are `aime` and `gpqa`.