| @@ -0,0 +1,885 @@ | |||
| # lq_token_test Implementation Plan | |||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | |||
| **Goal:** Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond. | |||
| **Architecture:** Keep the project as one binary crate with focused internal modules: `cli`, `config`, `runner`, `protocols`, `benchmarks`, and `metrics`. Datasets are not committed; the CLI downloads them into `data/benchmarks/`, writes metadata, and benchmark commands read only local files. | |||
| **Tech Stack:** Rust 2024, `clap`, `serde`, `serde_yaml`, `serde_json`, `tokio`, `reqwest` with Rustls, `anyhow`, `thiserror`, `tracing`, `indicatif`, `hdrhistogram`, `regex`, `sha2`, `chrono`, `futures`. | |||
| --- | |||
| ## File Structure | |||
| - Modify `Cargo.toml`: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies. | |||
| - Modify `.gitignore`: keep `target/` and add local benchmark data/output directories. | |||
| - Create `config.example.yaml`: example relay and benchmark configuration. | |||
| - Modify `src/main.rs`: async entry point, logging, command dispatch. | |||
| - Create `src/cli.rs`: `clap` command definitions. | |||
| - Create `src/config.rs`: YAML loading, provider lookup, environment expansion. | |||
| - Create `src/runner.rs`: provider-neutral request and response types plus request execution timing. | |||
| - Create `src/protocols/mod.rs`: protocol enum and dispatch. | |||
| - Create `src/protocols/openai.rs`: OpenAI `/chat/completions` request/response adapter. | |||
| - Create `src/protocols/anthropic.rs`: Anthropic `/v1/messages` request/response adapter. | |||
| - Create `src/benchmarks/mod.rs`: benchmark command orchestration and shared record types. | |||
| - Create `src/benchmarks/aime.rs`: AIME 2026 download, parse, prompt, judge integration. | |||
| - Create `src/benchmarks/gpqa.rs`: GPQA-Diamond download, parse, prompt, judge integration. | |||
| - Create `src/benchmarks/judge.rs`: numeric and multiple-choice answer extraction. | |||
| - Create `src/metrics.rs`: latency, success/error, and accuracy summaries. | |||
| ## Task 1: Dependencies And Project Skeleton | |||
| **Files:** | |||
| - Modify: `Cargo.toml` | |||
| - Modify: `.gitignore` | |||
| - Create: `config.example.yaml` | |||
| - Modify: `src/main.rs` | |||
| - Create: `src/cli.rs` | |||
| - Create: `src/config.rs` | |||
| - Create: `src/runner.rs` | |||
| - Create: `src/protocols/mod.rs` | |||
| - Create: `src/protocols/openai.rs` | |||
| - Create: `src/protocols/anthropic.rs` | |||
| - Create: `src/benchmarks/mod.rs` | |||
| - Create: `src/benchmarks/aime.rs` | |||
| - Create: `src/benchmarks/gpqa.rs` | |||
| - Create: `src/benchmarks/judge.rs` | |||
| - Create: `src/metrics.rs` | |||
| - [ ] **Step 1: Add crate dependencies** | |||
| Set `Cargo.toml` dependencies to: | |||
| ```toml | |||
| [package] | |||
| name = "lq_token_test" | |||
| version = "0.1.0" | |||
| edition = "2024" | |||
| [dependencies] | |||
| anyhow = "1" | |||
| chrono = { version = "0.4", features = ["serde"] } | |||
| clap = { version = "4", features = ["derive"] } | |||
| futures = "0.3" | |||
| hdrhistogram = "7" | |||
| indicatif = "0.17" | |||
| regex = "1" | |||
| reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } | |||
| serde = { version = "1", features = ["derive"] } | |||
| serde_json = "1" | |||
| serde_yaml = "0.9" | |||
| sha2 = "0.10" | |||
| thiserror = "2" | |||
| tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] } | |||
| tracing = "0.1" | |||
| tracing-subscriber = { version = "0.3", features = ["env-filter"] } | |||
| url = "2" | |||
| [dev-dependencies] | |||
| tempfile = "3" | |||
| wiremock = "0.6" | |||
| ``` | |||
| - [ ] **Step 2: Ignore local generated data** | |||
| Set `.gitignore` to: | |||
| ```gitignore | |||
| /target | |||
| /data/benchmarks | |||
| /reports | |||
| ``` | |||
| - [ ] **Step 3: Add example config** | |||
| Create `config.example.yaml`: | |||
| ```yaml | |||
| default_provider: openai | |||
| providers: | |||
| openai: | |||
| protocol: openai | |||
| base_url: "https://relay.example.com/v1" | |||
| api_token: "${OPENAI_RELAY_TOKEN}" | |||
| default_model: "gpt-4o-mini" | |||
| anthropic: | |||
| protocol: anthropic | |||
| base_url: "https://relay.example.com" | |||
| api_token: "${ANTHROPIC_RELAY_TOKEN}" | |||
| default_model: "claude-3-5-sonnet-latest" | |||
| benchmarks: | |||
| data_dir: "data/benchmarks" | |||
| aime2026: | |||
| source: "huggingface:MathArena/aime_2026" | |||
| split: "train" | |||
| gpqa_diamond: | |||
| source: "huggingface:Idavidrein/gpqa" | |||
| split: "gpqa_diamond" | |||
| ``` | |||
| - [ ] **Step 4: Create empty module declarations** | |||
| Replace `src/main.rs` with: | |||
| ```rust | |||
| mod benchmarks; | |||
| mod cli; | |||
| mod config; | |||
| mod metrics; | |||
| mod protocols; | |||
| mod runner; | |||
| use anyhow::Result; | |||
| use clap::Parser; | |||
| use cli::Cli; | |||
| #[tokio::main] | |||
| async fn main() -> Result<()> { | |||
| tracing_subscriber::fmt() | |||
| .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) | |||
| .init(); | |||
| let cli = Cli::parse(); | |||
| cli::dispatch(cli).await | |||
| } | |||
| ``` | |||
| Create placeholder modules that compile: | |||
| ```rust | |||
| // src/cli.rs | |||
| use anyhow::Result; | |||
| use clap::{Parser, Subcommand}; | |||
| use std::path::PathBuf; | |||
| #[derive(Debug, Parser)] | |||
| #[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")] | |||
| pub struct Cli { | |||
| #[command(subcommand)] | |||
| pub command: Command, | |||
| } | |||
| #[derive(Debug, Subcommand)] | |||
| pub enum Command { | |||
| Check { | |||
| #[arg(long, default_value = "config.yaml")] | |||
| config: PathBuf, | |||
| #[arg(long)] | |||
| provider: Option<String>, | |||
| #[arg(long)] | |||
| model: Option<String>, | |||
| #[arg(long)] | |||
| prompt: String, | |||
| }, | |||
| Dataset { | |||
| #[command(subcommand)] | |||
| command: DatasetCommand, | |||
| }, | |||
| Bench { | |||
| #[command(subcommand)] | |||
| command: BenchCommand, | |||
| }, | |||
| Rpm { | |||
| #[arg(long, default_value = "config.yaml")] | |||
| config: PathBuf, | |||
| #[arg(long)] | |||
| provider: Option<String>, | |||
| #[arg(long)] | |||
| model: Option<String>, | |||
| #[arg(long)] | |||
| rpm: u32, | |||
| #[arg(long)] | |||
| duration: String, | |||
| #[arg(long)] | |||
| prompt: String, | |||
| }, | |||
| } | |||
| #[derive(Debug, Subcommand)] | |||
| pub enum DatasetCommand { | |||
| Fetch { dataset: String }, | |||
| } | |||
| #[derive(Debug, Subcommand)] | |||
| pub enum BenchCommand { | |||
| Aime2026 { | |||
| #[arg(long, default_value = "config.yaml")] | |||
| config: PathBuf, | |||
| #[arg(long)] | |||
| provider: Option<String>, | |||
| #[arg(long)] | |||
| model: Option<String>, | |||
| #[arg(long, default_value_t = 4)] | |||
| concurrency: usize, | |||
| #[arg(long)] | |||
| limit: Option<usize>, | |||
| }, | |||
| GpqaDiamond { | |||
| #[arg(long, default_value = "config.yaml")] | |||
| config: PathBuf, | |||
| #[arg(long)] | |||
| provider: Option<String>, | |||
| #[arg(long)] | |||
| model: Option<String>, | |||
| #[arg(long, default_value_t = 4)] | |||
| concurrency: usize, | |||
| #[arg(long)] | |||
| limit: Option<usize>, | |||
| }, | |||
| } | |||
| pub async fn dispatch(cli: Cli) -> Result<()> { | |||
| match cli.command { | |||
| Command::Check { .. } => anyhow::bail!("check is not implemented yet"), | |||
| Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"), | |||
| Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"), | |||
| Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"), | |||
| } | |||
| } | |||
| ``` | |||
| Create each other module with one line: | |||
| ```rust | |||
| // src/config.rs, src/runner.rs, src/metrics.rs | |||
| ``` | |||
| For directories: | |||
| ```rust | |||
| // src/protocols/mod.rs | |||
| pub mod anthropic; | |||
| pub mod openai; | |||
| ``` | |||
| ```rust | |||
| // src/protocols/openai.rs | |||
| ``` | |||
| ```rust | |||
| // src/protocols/anthropic.rs | |||
| ``` | |||
| ```rust | |||
| // src/benchmarks/mod.rs | |||
| pub mod aime; | |||
| pub mod gpqa; | |||
| pub mod judge; | |||
| ``` | |||
| ```rust | |||
| // src/benchmarks/aime.rs | |||
| ``` | |||
| ```rust | |||
| // src/benchmarks/gpqa.rs | |||
| ``` | |||
| ```rust | |||
| // src/benchmarks/judge.rs | |||
| ``` | |||
| - [ ] **Step 5: Verify skeleton** | |||
| Run: `cargo fmt` | |||
| Run: `cargo test` | |||
| Expected: build succeeds with no tests or only placeholder warnings. | |||
| - [ ] **Step 6: Commit** | |||
| ```bash | |||
| git add Cargo.toml Cargo.lock .gitignore config.example.yaml src | |||
| git commit -m "chore: initialize cli project skeleton" | |||
| ``` | |||
| ## Task 2: Config Loading | |||
| **Files:** | |||
| - Modify: `src/config.rs` | |||
| - Modify: `src/cli.rs` | |||
| - [ ] **Step 1: Write config tests** | |||
| Add tests to `src/config.rs`: | |||
| ```rust | |||
| #[cfg(test)] | |||
| mod tests { | |||
| use super::*; | |||
| use std::fs; | |||
| #[test] | |||
| fn loads_provider_and_expands_env_token() { | |||
| unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") }; | |||
| let dir = tempfile::tempdir().unwrap(); | |||
| let path = dir.path().join("config.yaml"); | |||
| fs::write( | |||
| &path, | |||
| r#" | |||
| default_provider: openai | |||
| providers: | |||
| openai: | |||
| protocol: openai | |||
| base_url: "https://example.com/v1" | |||
| api_token: "${LQ_TEST_TOKEN}" | |||
| default_model: "gpt-test" | |||
| benchmarks: | |||
| data_dir: "data/benchmarks" | |||
| "#, | |||
| ) | |||
| .unwrap(); | |||
| let config = AppConfig::load(&path).unwrap(); | |||
| let provider = config.provider(None).unwrap(); | |||
| assert_eq!(provider.api_token, "secret-token"); | |||
| assert_eq!(provider.default_model, "gpt-test"); | |||
| assert_eq!(provider.protocol, ProtocolKind::Openai); | |||
| } | |||
| #[test] | |||
| fn rejects_missing_env_token() { | |||
| unsafe { std::env::remove_var("LQ_MISSING_TOKEN") }; | |||
| let dir = tempfile::tempdir().unwrap(); | |||
| let path = dir.path().join("config.yaml"); | |||
| fs::write( | |||
| &path, | |||
| r#" | |||
| default_provider: openai | |||
| providers: | |||
| openai: | |||
| protocol: openai | |||
| base_url: "https://example.com/v1" | |||
| api_token: "${LQ_MISSING_TOKEN}" | |||
| default_model: "gpt-test" | |||
| "#, | |||
| ) | |||
| .unwrap(); | |||
| let err = AppConfig::load(&path).unwrap_err().to_string(); | |||
| assert!(err.contains("LQ_MISSING_TOKEN")); | |||
| } | |||
| } | |||
| ``` | |||
| - [ ] **Step 2: Run tests to verify failure** | |||
| Run: `cargo test config::tests` | |||
| Expected: fails because `AppConfig` and `ProtocolKind` are not implemented. | |||
| - [ ] **Step 3: Implement config** | |||
| Implement `src/config.rs`: | |||
| ```rust | |||
| use serde::Deserialize; | |||
| use std::{collections::BTreeMap, fs, path::Path}; | |||
| use thiserror::Error; | |||
| #[derive(Debug, Error)] | |||
| pub enum ConfigError { | |||
| #[error("failed to read config {path}: {source}")] | |||
| Read { | |||
| path: String, | |||
| #[source] | |||
| source: std::io::Error, | |||
| }, | |||
| #[error("failed to parse yaml config: {0}")] | |||
| Parse(#[from] serde_yaml::Error), | |||
| #[error("unknown provider '{0}'")] | |||
| UnknownProvider(String), | |||
| #[error("config has no default_provider and no --provider was supplied")] | |||
| MissingProvider, | |||
| #[error("environment variable '{0}' referenced by config is not set")] | |||
| MissingEnv(String), | |||
| } | |||
| #[derive(Debug, Clone, Deserialize, PartialEq, Eq)] | |||
| #[serde(rename_all = "snake_case")] | |||
| pub enum ProtocolKind { | |||
| Openai, | |||
| Anthropic, | |||
| } | |||
| #[derive(Debug, Clone, Deserialize)] | |||
| pub struct ProviderConfig { | |||
| pub protocol: ProtocolKind, | |||
| pub base_url: String, | |||
| pub api_token: String, | |||
| pub default_model: String, | |||
| } | |||
| #[derive(Debug, Clone, Deserialize, Default)] | |||
| pub struct BenchmarkConfig { | |||
| #[serde(default = "default_data_dir")] | |||
| pub data_dir: String, | |||
| #[serde(default)] | |||
| pub aime2026: Option<DatasetConfig>, | |||
| #[serde(default)] | |||
| pub gpqa_diamond: Option<DatasetConfig>, | |||
| } | |||
| #[derive(Debug, Clone, Deserialize)] | |||
| pub struct DatasetConfig { | |||
| pub source: String, | |||
| pub split: String, | |||
| } | |||
| #[derive(Debug, Clone, Deserialize)] | |||
| pub struct AppConfig { | |||
| pub default_provider: Option<String>, | |||
| pub providers: BTreeMap<String, ProviderConfig>, | |||
| #[serde(default)] | |||
| pub benchmarks: BenchmarkConfig, | |||
| } | |||
| fn default_data_dir() -> String { | |||
| "data/benchmarks".to_string() | |||
| } | |||
| impl AppConfig { | |||
| pub fn load(path: &Path) -> Result<Self, ConfigError> { | |||
| let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read { | |||
| path: path.display().to_string(), | |||
| source, | |||
| })?; | |||
| let expanded = expand_env_refs(&raw)?; | |||
| Ok(serde_yaml::from_str(&expanded)?) | |||
| } | |||
| pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> { | |||
| let name = match provider { | |||
| Some(name) => name, | |||
| None => self | |||
| .default_provider | |||
| .as_deref() | |||
| .ok_or(ConfigError::MissingProvider)?, | |||
| }; | |||
| self.providers | |||
| .get(name) | |||
| .ok_or_else(|| ConfigError::UnknownProvider(name.to_string())) | |||
| } | |||
| } | |||
| fn expand_env_refs(input: &str) -> Result<String, ConfigError> { | |||
| let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex"); | |||
| let mut output = String::with_capacity(input.len()); | |||
| let mut last = 0; | |||
| for caps in re.captures_iter(input) { | |||
| let whole = caps.get(0).expect("whole match"); | |||
| let name = caps.get(1).expect("env name").as_str(); | |||
| output.push_str(&input[last..whole.start()]); | |||
| let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?; | |||
| output.push_str(&value); | |||
| last = whole.end(); | |||
| } | |||
| output.push_str(&input[last..]); | |||
| Ok(output) | |||
| } | |||
| ``` | |||
| - [ ] **Step 4: Run tests** | |||
| Run: `cargo test config::tests` | |||
| Expected: both config tests pass. | |||
| - [ ] **Step 5: Commit** | |||
| ```bash | |||
| git add src/config.rs src/cli.rs | |||
| git commit -m "feat: load yaml relay config" | |||
| ``` | |||
| ## Task 3: Dataset Download And Parsing | |||
| **Files:** | |||
| - Modify: `src/cli.rs` | |||
| - Modify: `src/benchmarks/mod.rs` | |||
| - Modify: `src/benchmarks/aime.rs` | |||
| - Modify: `src/benchmarks/gpqa.rs` | |||
| - Modify: `src/benchmarks/judge.rs` | |||
| - [ ] **Step 1: Add judge tests** | |||
| Add tests to `src/benchmarks/judge.rs`: | |||
| ```rust | |||
| #[cfg(test)] | |||
| mod tests { | |||
| use super::*; | |||
| #[test] | |||
| fn extracts_final_integer() { | |||
| assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string())); | |||
| assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string())); | |||
| } | |||
| #[test] | |||
| fn extracts_choice_letter() { | |||
| assert_eq!(extract_choice("Answer: C"), Some('C')); | |||
| assert_eq!(extract_choice("I choose (b)."), Some('B')); | |||
| } | |||
| } | |||
| ``` | |||
| - [ ] **Step 2: Implement judge functions** | |||
| Implement `src/benchmarks/judge.rs`: | |||
| ```rust | |||
| pub fn extract_final_integer(text: &str) -> Option<String> { | |||
| let re = regex::Regex::new(r"-?\d+").expect("valid integer regex"); | |||
| re.find_iter(text).last().map(|m| m.as_str().to_string()) | |||
| } | |||
| pub fn extract_choice(text: &str) -> Option<char> { | |||
| let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex"); | |||
| re.captures_iter(text) | |||
| .last() | |||
| .and_then(|caps| caps.get(1)) | |||
| .and_then(|m| m.as_str().chars().next()) | |||
| .map(|c| c.to_ascii_uppercase()) | |||
| } | |||
| pub fn judge_integer(output: &str, expected: &str) -> bool { | |||
| extract_final_integer(output).as_deref() == Some(expected.trim()) | |||
| } | |||
| pub fn judge_choice(output: &str, expected: char) -> bool { | |||
| extract_choice(output) == Some(expected.to_ascii_uppercase()) | |||
| } | |||
| ``` | |||
| - [ ] **Step 3: Add dataset parse tests with small fixtures** | |||
| Add tests in `src/benchmarks/aime.rs` and `src/benchmarks/gpqa.rs` using inline rows: | |||
| ```rust | |||
| #[cfg(test)] | |||
| mod tests { | |||
| use super::*; | |||
| #[test] | |||
| fn builds_aime_prompt() { | |||
| let case = AimeCase { | |||
| id: "1".to_string(), | |||
| problem: "What is 20 + 22?".to_string(), | |||
| answer: "42".to_string(), | |||
| }; | |||
| assert!(case.prompt().contains("What is 20 + 22?")); | |||
| assert!(case.prompt().contains("final integer answer")); | |||
| } | |||
| } | |||
| ``` | |||
| ```rust | |||
| #[cfg(test)] | |||
| mod tests { | |||
| use super::*; | |||
| #[test] | |||
| fn builds_gpqa_prompt() { | |||
| let case = GpqaCase { | |||
| id: "gpqa_1".to_string(), | |||
| question: "Which option is correct?".to_string(), | |||
| choices: [ | |||
| ("A".to_string(), "Alpha".to_string()), | |||
| ("B".to_string(), "Beta".to_string()), | |||
| ("C".to_string(), "Gamma".to_string()), | |||
| ("D".to_string(), "Delta".to_string()), | |||
| ], | |||
| answer: 'B', | |||
| }; | |||
| let prompt = case.prompt(); | |||
| assert!(prompt.contains("A. Alpha")); | |||
| assert!(prompt.contains("answer with exactly one letter")); | |||
| } | |||
| } | |||
| ``` | |||
| - [ ] **Step 4: Implement benchmark case types and fetch stubs** | |||
| Implement enough for prompts and later download wiring: | |||
| ```rust | |||
| // src/benchmarks/aime.rs | |||
| #[derive(Debug, Clone)] | |||
| pub struct AimeCase { | |||
| pub id: String, | |||
| pub problem: String, | |||
| pub answer: String, | |||
| } | |||
| impl AimeCase { | |||
| pub fn prompt(&self) -> String { | |||
| format!( | |||
| "Solve the following AIME problem. Return only the final integer answer.\n\n{}", | |||
| self.problem | |||
| ) | |||
| } | |||
| } | |||
| ``` | |||
| ```rust | |||
| // src/benchmarks/gpqa.rs | |||
| #[derive(Debug, Clone)] | |||
| pub struct GpqaCase { | |||
| pub id: String, | |||
| pub question: String, | |||
| pub choices: [(String, String); 4], | |||
| pub answer: char, | |||
| } | |||
| impl GpqaCase { | |||
| pub fn prompt(&self) -> String { | |||
| format!( | |||
| "Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}", | |||
| self.question, | |||
| self.choices[0].0, | |||
| self.choices[0].1, | |||
| self.choices[1].0, | |||
| self.choices[1].1, | |||
| self.choices[2].0, | |||
| self.choices[2].1, | |||
| self.choices[3].0, | |||
| self.choices[3].1 | |||
| ) | |||
| } | |||
| } | |||
| ``` | |||
| - [ ] **Step 5: Implement `dataset fetch`** | |||
| Add download functions that use Hugging Face raw URLs: | |||
| ```rust | |||
| // src/benchmarks/mod.rs | |||
| use anyhow::{Context, Result}; | |||
| use chrono::Utc; | |||
| use sha2::{Digest, Sha256}; | |||
| use std::{fs, path::{Path, PathBuf}}; | |||
| pub mod aime; | |||
| pub mod gpqa; | |||
| pub mod judge; | |||
| pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> { | |||
| match dataset { | |||
| "aime2026" => fetch_to_dir( | |||
| "aime2026", | |||
| "https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet", | |||
| data_dir, | |||
| None, | |||
| ) | |||
| .await, | |||
| "gpqa-diamond" => fetch_to_dir( | |||
| "gpqa_diamond", | |||
| "https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv", | |||
| data_dir, | |||
| std::env::var("HF_TOKEN").ok(), | |||
| ) | |||
| .await, | |||
| other => anyhow::bail!("unknown dataset '{other}'"), | |||
| } | |||
| } | |||
| async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> { | |||
| let dir = data_dir.join(name); | |||
| fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?; | |||
| let filename = url.rsplit('/').next().unwrap_or("dataset"); | |||
| let output = dir.join(filename); | |||
| let client = reqwest::Client::new(); | |||
| let mut request = client.get(url); | |||
| if let Some(token) = token { | |||
| request = request.bearer_auth(token); | |||
| } | |||
| let bytes = request.send().await?.error_for_status()?.bytes().await?; | |||
| fs::write(&output, &bytes)?; | |||
| let hash = Sha256::digest(&bytes); | |||
| let metadata = format!( | |||
| "name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n", | |||
| Utc::now().to_rfc3339(), | |||
| hash, | |||
| bytes.len() | |||
| ); | |||
| fs::write(dir.join("metadata.yaml"), metadata)?; | |||
| Ok(output) | |||
| } | |||
| ``` | |||
| - [ ] **Step 6: Wire dataset command** | |||
| Update `src/cli.rs` dispatch for `DatasetCommand::Fetch` to load default data dir from config when `config.yaml` exists, otherwise use `data/benchmarks`. | |||
| - [ ] **Step 7: Run tests** | |||
| Run: `cargo test benchmarks` | |||
| Expected: judge and prompt tests pass. Network fetch is not tested by default. | |||
| - [ ] **Step 8: Commit** | |||
| ```bash | |||
| git add src/benchmarks src/cli.rs | |||
| git commit -m "feat: add benchmark dataset fetch and prompts" | |||
| ``` | |||
| ## Task 4: Protocol Adapters And Single Check | |||
| **Files:** | |||
| - Modify: `src/runner.rs` | |||
| - Modify: `src/protocols/mod.rs` | |||
| - Modify: `src/protocols/openai.rs` | |||
| - Modify: `src/protocols/anthropic.rs` | |||
| - Modify: `src/cli.rs` | |||
| - [ ] **Step 1: Write HTTP adapter tests with wiremock** | |||
| Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text. | |||
| - [ ] **Step 2: Define runner types** | |||
| Implement provider-neutral types: | |||
| ```rust | |||
| #[derive(Debug, Clone)] | |||
| pub struct ModelRequest { | |||
| pub base_url: String, | |||
| pub api_token: String, | |||
| pub model: String, | |||
| pub prompt: String, | |||
| pub temperature: f32, | |||
| pub max_tokens: u32, | |||
| } | |||
| #[derive(Debug, Clone)] | |||
| pub struct ModelResponse { | |||
| pub text: String, | |||
| pub status: u16, | |||
| pub elapsed_ms: u128, | |||
| } | |||
| ``` | |||
| - [ ] **Step 3: Implement OpenAI adapter** | |||
| Post to `{base_url}/chat/completions` with `Authorization: Bearer <token>` and parse `choices[0].message.content`. | |||
| - [ ] **Step 4: Implement Anthropic adapter** | |||
| Post to `{base_url}/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and parse the first text block in `content`. | |||
| - [ ] **Step 5: Wire `check` command** | |||
| Load config, resolve provider and model, run one request, then print status, elapsed time, and response text. | |||
| - [ ] **Step 6: Run tests** | |||
| Run: `cargo test protocols runner` | |||
| Expected: mocked protocol tests pass. | |||
| - [ ] **Step 7: Commit** | |||
| ```bash | |||
| git add src/runner.rs src/protocols src/cli.rs | |||
| git commit -m "feat: add openai and anthropic request adapters" | |||
| ``` | |||
| ## Task 5: Metrics, Benchmark Runs, And RPM | |||
| **Files:** | |||
| - Modify: `src/metrics.rs` | |||
| - Modify: `src/benchmarks/mod.rs` | |||
| - Modify: `src/benchmarks/aime.rs` | |||
| - Modify: `src/benchmarks/gpqa.rs` | |||
| - Modify: `src/cli.rs` | |||
| - [ ] **Step 1: Add metrics tests** | |||
| Test success/error counts, accuracy, and latency percentile calculation. | |||
| - [ ] **Step 2: Implement metrics summary** | |||
| Use `hdrhistogram::Histogram<u64>` for elapsed milliseconds and counters for success, failure, correct, and total judged. | |||
| - [ ] **Step 3: Implement local benchmark loading** | |||
| For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch <name>`. | |||
| - [ ] **Step 4: Implement benchmark execution** | |||
| Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics. | |||
| - [ ] **Step 5: Implement RPM command** | |||
| Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary. | |||
| - [ ] **Step 6: Run tests** | |||
| Run: `cargo test` | |||
| Expected: unit tests pass. | |||
| - [ ] **Step 7: Commit** | |||
| ```bash | |||
| git add src/metrics.rs src/benchmarks src/cli.rs | |||
| git commit -m "feat: run benchmark and rpm tests" | |||
| ``` | |||
| ## Task 6: Verification And Docs | |||
| **Files:** | |||
| - Create: `README.md` | |||
| - Modify: `config.example.yaml` | |||
| - [ ] **Step 1: Document usage** | |||
| Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test. | |||
| - [ ] **Step 2: Verify formatting and tests** | |||
| Run: | |||
| ```bash | |||
| cargo fmt --check | |||
| cargo test | |||
| cargo clippy --all-targets -- -D warnings | |||
| ``` | |||
| Expected: all pass. | |||
| - [ ] **Step 3: Manual smoke commands** | |||
| Run: | |||
| ```bash | |||
| cargo run -- --help | |||
| cargo run -- dataset fetch aime2026 | |||
| ``` | |||
| Expected: help renders; AIME fetch creates `data/benchmarks/aime2026/metadata.yaml`. | |||
| - [ ] **Step 4: Commit** | |||
| ```bash | |||
| git add README.md config.example.yaml | |||
| git commit -m "docs: add usage guide" | |||
| ``` | |||
| ## Self-Review | |||
| - Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage. | |||
| - Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior. | |||
| - Type consistency: dataset names are `aime2026` and `gpqa-diamond` in CLI; local directory names are `aime2026` and `gpqa_diamond`; module names are `aime` and `gpqa`. | |||