| @@ -0,0 +1,885 @@ | |||||
| # lq_token_test Implementation Plan | |||||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | |||||
| **Goal:** Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond. | |||||
| **Architecture:** Keep the project as one binary crate with focused internal modules: `cli`, `config`, `runner`, `protocols`, `benchmarks`, and `metrics`. Datasets are not committed; the CLI downloads them into `data/benchmarks/`, writes metadata, and benchmark commands read only local files. | |||||
| **Tech Stack:** Rust 2024, `clap`, `serde`, `serde_yaml`, `serde_json`, `tokio`, `reqwest` with Rustls, `anyhow`, `thiserror`, `tracing`, `indicatif`, `hdrhistogram`, `regex`, `sha2`, `chrono`, `futures`. | |||||
| --- | |||||
| ## File Structure | |||||
| - Modify `Cargo.toml`: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies. | |||||
| - Modify `.gitignore`: keep `target/` and add local benchmark data/output directories. | |||||
| - Create `config.example.yaml`: example relay and benchmark configuration. | |||||
| - Modify `src/main.rs`: async entry point, logging, command dispatch. | |||||
| - Create `src/cli.rs`: `clap` command definitions. | |||||
| - Create `src/config.rs`: YAML loading, provider lookup, environment expansion. | |||||
| - Create `src/runner.rs`: provider-neutral request and response types plus request execution timing. | |||||
| - Create `src/protocols/mod.rs`: protocol enum and dispatch. | |||||
| - Create `src/protocols/openai.rs`: OpenAI `/chat/completions` request/response adapter. | |||||
| - Create `src/protocols/anthropic.rs`: Anthropic `/v1/messages` request/response adapter. | |||||
| - Create `src/benchmarks/mod.rs`: benchmark command orchestration and shared record types. | |||||
| - Create `src/benchmarks/aime.rs`: AIME 2026 download, parse, prompt, judge integration. | |||||
| - Create `src/benchmarks/gpqa.rs`: GPQA-Diamond download, parse, prompt, judge integration. | |||||
| - Create `src/benchmarks/judge.rs`: numeric and multiple-choice answer extraction. | |||||
| - Create `src/metrics.rs`: latency, success/error, and accuracy summaries. | |||||
| ## Task 1: Dependencies And Project Skeleton | |||||
| **Files:** | |||||
| - Modify: `Cargo.toml` | |||||
| - Modify: `.gitignore` | |||||
| - Create: `config.example.yaml` | |||||
| - Modify: `src/main.rs` | |||||
| - Create: `src/cli.rs` | |||||
| - Create: `src/config.rs` | |||||
| - Create: `src/runner.rs` | |||||
| - Create: `src/protocols/mod.rs` | |||||
| - Create: `src/protocols/openai.rs` | |||||
| - Create: `src/protocols/anthropic.rs` | |||||
| - Create: `src/benchmarks/mod.rs` | |||||
| - Create: `src/benchmarks/aime.rs` | |||||
| - Create: `src/benchmarks/gpqa.rs` | |||||
| - Create: `src/benchmarks/judge.rs` | |||||
| - Create: `src/metrics.rs` | |||||
| - [ ] **Step 1: Add crate dependencies** | |||||
| Set `Cargo.toml` dependencies to: | |||||
| ```toml | |||||
| [package] | |||||
| name = "lq_token_test" | |||||
| version = "0.1.0" | |||||
| edition = "2024" | |||||
| [dependencies] | |||||
| anyhow = "1" | |||||
| chrono = { version = "0.4", features = ["serde"] } | |||||
| clap = { version = "4", features = ["derive"] } | |||||
| futures = "0.3" | |||||
| hdrhistogram = "7" | |||||
| indicatif = "0.17" | |||||
| regex = "1" | |||||
| reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } | |||||
| serde = { version = "1", features = ["derive"] } | |||||
| serde_json = "1" | |||||
| serde_yaml = "0.9" | |||||
| sha2 = "0.10" | |||||
| thiserror = "2" | |||||
| tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] } | |||||
| tracing = "0.1" | |||||
| tracing-subscriber = { version = "0.3", features = ["env-filter"] } | |||||
| url = "2" | |||||
| [dev-dependencies] | |||||
| tempfile = "3" | |||||
| wiremock = "0.6" | |||||
| ``` | |||||
| - [ ] **Step 2: Ignore local generated data** | |||||
| Set `.gitignore` to: | |||||
| ```gitignore | |||||
| /target | |||||
| /data/benchmarks | |||||
| /reports | |||||
| ``` | |||||
| - [ ] **Step 3: Add example config** | |||||
| Create `config.example.yaml`: | |||||
| ```yaml | |||||
| default_provider: openai | |||||
| providers: | |||||
| openai: | |||||
| protocol: openai | |||||
| base_url: "https://relay.example.com/v1" | |||||
| api_token: "${OPENAI_RELAY_TOKEN}" | |||||
| default_model: "gpt-4o-mini" | |||||
| anthropic: | |||||
| protocol: anthropic | |||||
| base_url: "https://relay.example.com" | |||||
| api_token: "${ANTHROPIC_RELAY_TOKEN}" | |||||
| default_model: "claude-3-5-sonnet-latest" | |||||
| benchmarks: | |||||
| data_dir: "data/benchmarks" | |||||
| aime2026: | |||||
| source: "huggingface:MathArena/aime_2026" | |||||
| split: "train" | |||||
| gpqa_diamond: | |||||
| source: "huggingface:Idavidrein/gpqa" | |||||
| split: "gpqa_diamond" | |||||
| ``` | |||||
| - [ ] **Step 4: Create empty module declarations** | |||||
| Replace `src/main.rs` with: | |||||
| ```rust | |||||
| mod benchmarks; | |||||
| mod cli; | |||||
| mod config; | |||||
| mod metrics; | |||||
| mod protocols; | |||||
| mod runner; | |||||
| use anyhow::Result; | |||||
| use clap::Parser; | |||||
| use cli::Cli; | |||||
| #[tokio::main] | |||||
| async fn main() -> Result<()> { | |||||
| tracing_subscriber::fmt() | |||||
| .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) | |||||
| .init(); | |||||
| let cli = Cli::parse(); | |||||
| cli::dispatch(cli).await | |||||
| } | |||||
| ``` | |||||
| Create placeholder modules that compile: | |||||
| ```rust | |||||
| // src/cli.rs | |||||
| use anyhow::Result; | |||||
| use clap::{Parser, Subcommand}; | |||||
| use std::path::PathBuf; | |||||
| #[derive(Debug, Parser)] | |||||
| #[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")] | |||||
| pub struct Cli { | |||||
| #[command(subcommand)] | |||||
| pub command: Command, | |||||
| } | |||||
| #[derive(Debug, Subcommand)] | |||||
| pub enum Command { | |||||
| Check { | |||||
| #[arg(long, default_value = "config.yaml")] | |||||
| config: PathBuf, | |||||
| #[arg(long)] | |||||
| provider: Option<String>, | |||||
| #[arg(long)] | |||||
| model: Option<String>, | |||||
| #[arg(long)] | |||||
| prompt: String, | |||||
| }, | |||||
| Dataset { | |||||
| #[command(subcommand)] | |||||
| command: DatasetCommand, | |||||
| }, | |||||
| Bench { | |||||
| #[command(subcommand)] | |||||
| command: BenchCommand, | |||||
| }, | |||||
| Rpm { | |||||
| #[arg(long, default_value = "config.yaml")] | |||||
| config: PathBuf, | |||||
| #[arg(long)] | |||||
| provider: Option<String>, | |||||
| #[arg(long)] | |||||
| model: Option<String>, | |||||
| #[arg(long)] | |||||
| rpm: u32, | |||||
| #[arg(long)] | |||||
| duration: String, | |||||
| #[arg(long)] | |||||
| prompt: String, | |||||
| }, | |||||
| } | |||||
| #[derive(Debug, Subcommand)] | |||||
| pub enum DatasetCommand { | |||||
| Fetch { dataset: String }, | |||||
| } | |||||
| #[derive(Debug, Subcommand)] | |||||
| pub enum BenchCommand { | |||||
| Aime2026 { | |||||
| #[arg(long, default_value = "config.yaml")] | |||||
| config: PathBuf, | |||||
| #[arg(long)] | |||||
| provider: Option<String>, | |||||
| #[arg(long)] | |||||
| model: Option<String>, | |||||
| #[arg(long, default_value_t = 4)] | |||||
| concurrency: usize, | |||||
| #[arg(long)] | |||||
| limit: Option<usize>, | |||||
| }, | |||||
| GpqaDiamond { | |||||
| #[arg(long, default_value = "config.yaml")] | |||||
| config: PathBuf, | |||||
| #[arg(long)] | |||||
| provider: Option<String>, | |||||
| #[arg(long)] | |||||
| model: Option<String>, | |||||
| #[arg(long, default_value_t = 4)] | |||||
| concurrency: usize, | |||||
| #[arg(long)] | |||||
| limit: Option<usize>, | |||||
| }, | |||||
| } | |||||
| pub async fn dispatch(cli: Cli) -> Result<()> { | |||||
| match cli.command { | |||||
| Command::Check { .. } => anyhow::bail!("check is not implemented yet"), | |||||
| Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"), | |||||
| Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"), | |||||
| Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"), | |||||
| } | |||||
| } | |||||
| ``` | |||||
| Create each other module with one line: | |||||
| ```rust | |||||
| // src/config.rs, src/runner.rs, src/metrics.rs | |||||
| ``` | |||||
| For directories: | |||||
| ```rust | |||||
| // src/protocols/mod.rs | |||||
| pub mod anthropic; | |||||
| pub mod openai; | |||||
| ``` | |||||
| ```rust | |||||
| // src/protocols/openai.rs | |||||
| ``` | |||||
| ```rust | |||||
| // src/protocols/anthropic.rs | |||||
| ``` | |||||
| ```rust | |||||
| // src/benchmarks/mod.rs | |||||
| pub mod aime; | |||||
| pub mod gpqa; | |||||
| pub mod judge; | |||||
| ``` | |||||
| ```rust | |||||
| // src/benchmarks/aime.rs | |||||
| ``` | |||||
| ```rust | |||||
| // src/benchmarks/gpqa.rs | |||||
| ``` | |||||
| ```rust | |||||
| // src/benchmarks/judge.rs | |||||
| ``` | |||||
| - [ ] **Step 5: Verify skeleton** | |||||
| Run: `cargo fmt` | |||||
| Run: `cargo test` | |||||
| Expected: build succeeds with no tests or only placeholder warnings. | |||||
| - [ ] **Step 6: Commit** | |||||
| ```bash | |||||
| git add Cargo.toml Cargo.lock .gitignore config.example.yaml src | |||||
| git commit -m "chore: initialize cli project skeleton" | |||||
| ``` | |||||
| ## Task 2: Config Loading | |||||
| **Files:** | |||||
| - Modify: `src/config.rs` | |||||
| - Modify: `src/cli.rs` | |||||
| - [ ] **Step 1: Write config tests** | |||||
| Add tests to `src/config.rs`: | |||||
| ```rust | |||||
| #[cfg(test)] | |||||
| mod tests { | |||||
| use super::*; | |||||
| use std::fs; | |||||
| #[test] | |||||
| fn loads_provider_and_expands_env_token() { | |||||
| unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") }; | |||||
| let dir = tempfile::tempdir().unwrap(); | |||||
| let path = dir.path().join("config.yaml"); | |||||
| fs::write( | |||||
| &path, | |||||
| r#" | |||||
| default_provider: openai | |||||
| providers: | |||||
| openai: | |||||
| protocol: openai | |||||
| base_url: "https://example.com/v1" | |||||
| api_token: "${LQ_TEST_TOKEN}" | |||||
| default_model: "gpt-test" | |||||
| benchmarks: | |||||
| data_dir: "data/benchmarks" | |||||
| "#, | |||||
| ) | |||||
| .unwrap(); | |||||
| let config = AppConfig::load(&path).unwrap(); | |||||
| let provider = config.provider(None).unwrap(); | |||||
| assert_eq!(provider.api_token, "secret-token"); | |||||
| assert_eq!(provider.default_model, "gpt-test"); | |||||
| assert_eq!(provider.protocol, ProtocolKind::Openai); | |||||
| } | |||||
| #[test] | |||||
| fn rejects_missing_env_token() { | |||||
| unsafe { std::env::remove_var("LQ_MISSING_TOKEN") }; | |||||
| let dir = tempfile::tempdir().unwrap(); | |||||
| let path = dir.path().join("config.yaml"); | |||||
| fs::write( | |||||
| &path, | |||||
| r#" | |||||
| default_provider: openai | |||||
| providers: | |||||
| openai: | |||||
| protocol: openai | |||||
| base_url: "https://example.com/v1" | |||||
| api_token: "${LQ_MISSING_TOKEN}" | |||||
| default_model: "gpt-test" | |||||
| "#, | |||||
| ) | |||||
| .unwrap(); | |||||
| let err = AppConfig::load(&path).unwrap_err().to_string(); | |||||
| assert!(err.contains("LQ_MISSING_TOKEN")); | |||||
| } | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 2: Run tests to verify failure** | |||||
| Run: `cargo test config::tests` | |||||
| Expected: fails because `AppConfig` and `ProtocolKind` are not implemented. | |||||
| - [ ] **Step 3: Implement config** | |||||
| Implement `src/config.rs`: | |||||
| ```rust | |||||
| use serde::Deserialize; | |||||
| use std::{collections::BTreeMap, fs, path::Path}; | |||||
| use thiserror::Error; | |||||
| #[derive(Debug, Error)] | |||||
| pub enum ConfigError { | |||||
| #[error("failed to read config {path}: {source}")] | |||||
| Read { | |||||
| path: String, | |||||
| #[source] | |||||
| source: std::io::Error, | |||||
| }, | |||||
| #[error("failed to parse yaml config: {0}")] | |||||
| Parse(#[from] serde_yaml::Error), | |||||
| #[error("unknown provider '{0}'")] | |||||
| UnknownProvider(String), | |||||
| #[error("config has no default_provider and no --provider was supplied")] | |||||
| MissingProvider, | |||||
| #[error("environment variable '{0}' referenced by config is not set")] | |||||
| MissingEnv(String), | |||||
| } | |||||
| #[derive(Debug, Clone, Deserialize, PartialEq, Eq)] | |||||
| #[serde(rename_all = "snake_case")] | |||||
| pub enum ProtocolKind { | |||||
| Openai, | |||||
| Anthropic, | |||||
| } | |||||
| #[derive(Debug, Clone, Deserialize)] | |||||
| pub struct ProviderConfig { | |||||
| pub protocol: ProtocolKind, | |||||
| pub base_url: String, | |||||
| pub api_token: String, | |||||
| pub default_model: String, | |||||
| } | |||||
| #[derive(Debug, Clone, Deserialize, Default)] | |||||
| pub struct BenchmarkConfig { | |||||
| #[serde(default = "default_data_dir")] | |||||
| pub data_dir: String, | |||||
| #[serde(default)] | |||||
| pub aime2026: Option<DatasetConfig>, | |||||
| #[serde(default)] | |||||
| pub gpqa_diamond: Option<DatasetConfig>, | |||||
| } | |||||
| #[derive(Debug, Clone, Deserialize)] | |||||
| pub struct DatasetConfig { | |||||
| pub source: String, | |||||
| pub split: String, | |||||
| } | |||||
| #[derive(Debug, Clone, Deserialize)] | |||||
| pub struct AppConfig { | |||||
| pub default_provider: Option<String>, | |||||
| pub providers: BTreeMap<String, ProviderConfig>, | |||||
| #[serde(default)] | |||||
| pub benchmarks: BenchmarkConfig, | |||||
| } | |||||
| fn default_data_dir() -> String { | |||||
| "data/benchmarks".to_string() | |||||
| } | |||||
| impl AppConfig { | |||||
| pub fn load(path: &Path) -> Result<Self, ConfigError> { | |||||
| let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read { | |||||
| path: path.display().to_string(), | |||||
| source, | |||||
| })?; | |||||
| let expanded = expand_env_refs(&raw)?; | |||||
| Ok(serde_yaml::from_str(&expanded)?) | |||||
| } | |||||
| pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> { | |||||
| let name = match provider { | |||||
| Some(name) => name, | |||||
| None => self | |||||
| .default_provider | |||||
| .as_deref() | |||||
| .ok_or(ConfigError::MissingProvider)?, | |||||
| }; | |||||
| self.providers | |||||
| .get(name) | |||||
| .ok_or_else(|| ConfigError::UnknownProvider(name.to_string())) | |||||
| } | |||||
| } | |||||
| fn expand_env_refs(input: &str) -> Result<String, ConfigError> { | |||||
| let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex"); | |||||
| let mut output = String::with_capacity(input.len()); | |||||
| let mut last = 0; | |||||
| for caps in re.captures_iter(input) { | |||||
| let whole = caps.get(0).expect("whole match"); | |||||
| let name = caps.get(1).expect("env name").as_str(); | |||||
| output.push_str(&input[last..whole.start()]); | |||||
| let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?; | |||||
| output.push_str(&value); | |||||
| last = whole.end(); | |||||
| } | |||||
| output.push_str(&input[last..]); | |||||
| Ok(output) | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 4: Run tests** | |||||
| Run: `cargo test config::tests` | |||||
| Expected: both config tests pass. | |||||
| - [ ] **Step 5: Commit** | |||||
| ```bash | |||||
| git add src/config.rs src/cli.rs | |||||
| git commit -m "feat: load yaml relay config" | |||||
| ``` | |||||
| ## Task 3: Dataset Download And Parsing | |||||
| **Files:** | |||||
| - Modify: `src/cli.rs` | |||||
| - Modify: `src/benchmarks/mod.rs` | |||||
| - Modify: `src/benchmarks/aime.rs` | |||||
| - Modify: `src/benchmarks/gpqa.rs` | |||||
| - Modify: `src/benchmarks/judge.rs` | |||||
| - [ ] **Step 1: Add judge tests** | |||||
| Add tests to `src/benchmarks/judge.rs`: | |||||
| ```rust | |||||
| #[cfg(test)] | |||||
| mod tests { | |||||
| use super::*; | |||||
| #[test] | |||||
| fn extracts_final_integer() { | |||||
| assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string())); | |||||
| assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string())); | |||||
| } | |||||
| #[test] | |||||
| fn extracts_choice_letter() { | |||||
| assert_eq!(extract_choice("Answer: C"), Some('C')); | |||||
| assert_eq!(extract_choice("I choose (b)."), Some('B')); | |||||
| } | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 2: Implement judge functions** | |||||
| Implement `src/benchmarks/judge.rs`: | |||||
| ```rust | |||||
| pub fn extract_final_integer(text: &str) -> Option<String> { | |||||
| let re = regex::Regex::new(r"-?\d+").expect("valid integer regex"); | |||||
| re.find_iter(text).last().map(|m| m.as_str().to_string()) | |||||
| } | |||||
| pub fn extract_choice(text: &str) -> Option<char> { | |||||
| let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex"); | |||||
| re.captures_iter(text) | |||||
| .last() | |||||
| .and_then(|caps| caps.get(1)) | |||||
| .and_then(|m| m.as_str().chars().next()) | |||||
| .map(|c| c.to_ascii_uppercase()) | |||||
| } | |||||
| pub fn judge_integer(output: &str, expected: &str) -> bool { | |||||
| extract_final_integer(output).as_deref() == Some(expected.trim()) | |||||
| } | |||||
| pub fn judge_choice(output: &str, expected: char) -> bool { | |||||
| extract_choice(output) == Some(expected.to_ascii_uppercase()) | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 3: Add dataset parse tests with small fixtures** | |||||
| Add tests in `src/benchmarks/aime.rs` and `src/benchmarks/gpqa.rs` using inline rows: | |||||
| ```rust | |||||
| #[cfg(test)] | |||||
| mod tests { | |||||
| use super::*; | |||||
| #[test] | |||||
| fn builds_aime_prompt() { | |||||
| let case = AimeCase { | |||||
| id: "1".to_string(), | |||||
| problem: "What is 20 + 22?".to_string(), | |||||
| answer: "42".to_string(), | |||||
| }; | |||||
| assert!(case.prompt().contains("What is 20 + 22?")); | |||||
| assert!(case.prompt().contains("final integer answer")); | |||||
| } | |||||
| } | |||||
| ``` | |||||
| ```rust | |||||
| #[cfg(test)] | |||||
| mod tests { | |||||
| use super::*; | |||||
| #[test] | |||||
| fn builds_gpqa_prompt() { | |||||
| let case = GpqaCase { | |||||
| id: "gpqa_1".to_string(), | |||||
| question: "Which option is correct?".to_string(), | |||||
| choices: [ | |||||
| ("A".to_string(), "Alpha".to_string()), | |||||
| ("B".to_string(), "Beta".to_string()), | |||||
| ("C".to_string(), "Gamma".to_string()), | |||||
| ("D".to_string(), "Delta".to_string()), | |||||
| ], | |||||
| answer: 'B', | |||||
| }; | |||||
| let prompt = case.prompt(); | |||||
| assert!(prompt.contains("A. Alpha")); | |||||
| assert!(prompt.contains("answer with exactly one letter")); | |||||
| } | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 4: Implement benchmark case types and fetch stubs** | |||||
| Implement enough for prompts and later download wiring: | |||||
| ```rust | |||||
| // src/benchmarks/aime.rs | |||||
| #[derive(Debug, Clone)] | |||||
| pub struct AimeCase { | |||||
| pub id: String, | |||||
| pub problem: String, | |||||
| pub answer: String, | |||||
| } | |||||
| impl AimeCase { | |||||
| pub fn prompt(&self) -> String { | |||||
| format!( | |||||
| "Solve the following AIME problem. Return only the final integer answer.\n\n{}", | |||||
| self.problem | |||||
| ) | |||||
| } | |||||
| } | |||||
| ``` | |||||
| ```rust | |||||
| // src/benchmarks/gpqa.rs | |||||
| #[derive(Debug, Clone)] | |||||
| pub struct GpqaCase { | |||||
| pub id: String, | |||||
| pub question: String, | |||||
| pub choices: [(String, String); 4], | |||||
| pub answer: char, | |||||
| } | |||||
| impl GpqaCase { | |||||
| pub fn prompt(&self) -> String { | |||||
| format!( | |||||
| "Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}", | |||||
| self.question, | |||||
| self.choices[0].0, | |||||
| self.choices[0].1, | |||||
| self.choices[1].0, | |||||
| self.choices[1].1, | |||||
| self.choices[2].0, | |||||
| self.choices[2].1, | |||||
| self.choices[3].0, | |||||
| self.choices[3].1 | |||||
| ) | |||||
| } | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 5: Implement `dataset fetch`** | |||||
| Add download functions that use Hugging Face raw URLs: | |||||
| ```rust | |||||
| // src/benchmarks/mod.rs | |||||
| use anyhow::{Context, Result}; | |||||
| use chrono::Utc; | |||||
| use sha2::{Digest, Sha256}; | |||||
| use std::{fs, path::{Path, PathBuf}}; | |||||
| pub mod aime; | |||||
| pub mod gpqa; | |||||
| pub mod judge; | |||||
| pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> { | |||||
| match dataset { | |||||
| "aime2026" => fetch_to_dir( | |||||
| "aime2026", | |||||
| "https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet", | |||||
| data_dir, | |||||
| None, | |||||
| ) | |||||
| .await, | |||||
| "gpqa-diamond" => fetch_to_dir( | |||||
| "gpqa_diamond", | |||||
| "https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv", | |||||
| data_dir, | |||||
| std::env::var("HF_TOKEN").ok(), | |||||
| ) | |||||
| .await, | |||||
| other => anyhow::bail!("unknown dataset '{other}'"), | |||||
| } | |||||
| } | |||||
| async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> { | |||||
| let dir = data_dir.join(name); | |||||
| fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?; | |||||
| let filename = url.rsplit('/').next().unwrap_or("dataset"); | |||||
| let output = dir.join(filename); | |||||
| let client = reqwest::Client::new(); | |||||
| let mut request = client.get(url); | |||||
| if let Some(token) = token { | |||||
| request = request.bearer_auth(token); | |||||
| } | |||||
| let bytes = request.send().await?.error_for_status()?.bytes().await?; | |||||
| fs::write(&output, &bytes)?; | |||||
| let hash = Sha256::digest(&bytes); | |||||
| let metadata = format!( | |||||
| "name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n", | |||||
| Utc::now().to_rfc3339(), | |||||
| hash, | |||||
| bytes.len() | |||||
| ); | |||||
| fs::write(dir.join("metadata.yaml"), metadata)?; | |||||
| Ok(output) | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 6: Wire dataset command** | |||||
| Update `src/cli.rs` dispatch for `DatasetCommand::Fetch` to load default data dir from config when `config.yaml` exists, otherwise use `data/benchmarks`. | |||||
| - [ ] **Step 7: Run tests** | |||||
| Run: `cargo test benchmarks` | |||||
| Expected: judge and prompt tests pass. Network fetch is not tested by default. | |||||
| - [ ] **Step 8: Commit** | |||||
| ```bash | |||||
| git add src/benchmarks src/cli.rs | |||||
| git commit -m "feat: add benchmark dataset fetch and prompts" | |||||
| ``` | |||||
| ## Task 4: Protocol Adapters And Single Check | |||||
| **Files:** | |||||
| - Modify: `src/runner.rs` | |||||
| - Modify: `src/protocols/mod.rs` | |||||
| - Modify: `src/protocols/openai.rs` | |||||
| - Modify: `src/protocols/anthropic.rs` | |||||
| - Modify: `src/cli.rs` | |||||
| - [ ] **Step 1: Write HTTP adapter tests with wiremock** | |||||
| Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text. | |||||
| - [ ] **Step 2: Define runner types** | |||||
| Implement provider-neutral types: | |||||
| ```rust | |||||
| #[derive(Debug, Clone)] | |||||
| pub struct ModelRequest { | |||||
| pub base_url: String, | |||||
| pub api_token: String, | |||||
| pub model: String, | |||||
| pub prompt: String, | |||||
| pub temperature: f32, | |||||
| pub max_tokens: u32, | |||||
| } | |||||
| #[derive(Debug, Clone)] | |||||
| pub struct ModelResponse { | |||||
| pub text: String, | |||||
| pub status: u16, | |||||
| pub elapsed_ms: u128, | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 3: Implement OpenAI adapter** | |||||
| Post to `{base_url}/chat/completions` with `Authorization: Bearer <token>` and parse `choices[0].message.content`. | |||||
| - [ ] **Step 4: Implement Anthropic adapter** | |||||
| Post to `{base_url}/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and parse the first text block in `content`. | |||||
| - [ ] **Step 5: Wire `check` command** | |||||
| Load config, resolve provider and model, run one request, then print status, elapsed time, and response text. | |||||
| - [ ] **Step 6: Run tests** | |||||
| Run: `cargo test protocols runner` | |||||
| Expected: mocked protocol tests pass. | |||||
| - [ ] **Step 7: Commit** | |||||
| ```bash | |||||
| git add src/runner.rs src/protocols src/cli.rs | |||||
| git commit -m "feat: add openai and anthropic request adapters" | |||||
| ``` | |||||
| ## Task 5: Metrics, Benchmark Runs, And RPM | |||||
| **Files:** | |||||
| - Modify: `src/metrics.rs` | |||||
| - Modify: `src/benchmarks/mod.rs` | |||||
| - Modify: `src/benchmarks/aime.rs` | |||||
| - Modify: `src/benchmarks/gpqa.rs` | |||||
| - Modify: `src/cli.rs` | |||||
| - [ ] **Step 1: Add metrics tests** | |||||
| Test success/error counts, accuracy, and latency percentile calculation. | |||||
| - [ ] **Step 2: Implement metrics summary** | |||||
| Use `hdrhistogram::Histogram<u64>` for elapsed milliseconds and counters for success, failure, correct, and total judged. | |||||
| - [ ] **Step 3: Implement local benchmark loading** | |||||
| For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch <name>`. | |||||
| - [ ] **Step 4: Implement benchmark execution** | |||||
| Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics. | |||||
| - [ ] **Step 5: Implement RPM command** | |||||
| Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary. | |||||
| - [ ] **Step 6: Run tests** | |||||
| Run: `cargo test` | |||||
| Expected: unit tests pass. | |||||
| - [ ] **Step 7: Commit** | |||||
| ```bash | |||||
| git add src/metrics.rs src/benchmarks src/cli.rs | |||||
| git commit -m "feat: run benchmark and rpm tests" | |||||
| ``` | |||||
| ## Task 6: Verification And Docs | |||||
| **Files:** | |||||
| - Create: `README.md` | |||||
| - Modify: `config.example.yaml` | |||||
| - [ ] **Step 1: Document usage** | |||||
| Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test. | |||||
| - [ ] **Step 2: Verify formatting and tests** | |||||
| Run: | |||||
| ```bash | |||||
| cargo fmt --check | |||||
| cargo test | |||||
| cargo clippy --all-targets -- -D warnings | |||||
| ``` | |||||
| Expected: all pass. | |||||
| - [ ] **Step 3: Manual smoke commands** | |||||
| Run: | |||||
| ```bash | |||||
| cargo run -- --help | |||||
| cargo run -- dataset fetch aime2026 | |||||
| ``` | |||||
| Expected: help renders; AIME fetch creates `data/benchmarks/aime2026/metadata.yaml`. | |||||
| - [ ] **Step 4: Commit** | |||||
| ```bash | |||||
| git add README.md config.example.yaml | |||||
| git commit -m "docs: add usage guide" | |||||
| ``` | |||||
| ## Self-Review | |||||
| - Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage. | |||||
| - Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior. | |||||
| - Type consistency: dataset names are `aime2026` and `gpqa-diamond` in CLI; local directory names are `aime2026` and `gpqa_diamond`; module names are `aime` and `gpqa`. | |||||