For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond.
Architecture: Keep the project as one binary crate with focused internal modules: cli, config, runner, protocols, benchmarks, and metrics. Datasets are not committed; the CLI downloads them into data/benchmarks/, writes metadata, and benchmark commands read only local files.
Tech Stack: Rust 2024, clap, serde, serde_yaml, serde_json, tokio, reqwest with Rustls, anyhow, thiserror, tracing, indicatif, hdrhistogram, regex, sha2, chrono, futures.
Cargo.toml: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies..gitignore: keep target/ and add local benchmark data/output directories.config.example.yaml: example relay and benchmark configuration.src/main.rs: async entry point, logging, command dispatch.src/cli.rs: clap command definitions.src/config.rs: YAML loading, provider lookup, environment expansion.src/runner.rs: provider-neutral request and response types plus request execution timing.src/protocols/mod.rs: protocol enum and dispatch.src/protocols/openai.rs: OpenAI /chat/completions request/response adapter.src/protocols/anthropic.rs: Anthropic /v1/messages request/response adapter.src/benchmarks/mod.rs: benchmark command orchestration and shared record types.src/benchmarks/aime.rs: AIME 2026 download, parse, prompt, judge integration.src/benchmarks/gpqa.rs: GPQA-Diamond download, parse, prompt, judge integration.src/benchmarks/judge.rs: numeric and multiple-choice answer extraction.src/metrics.rs: latency, success/error, and accuracy summaries.Files:
Modify: Cargo.toml
Modify: .gitignore
Create: config.example.yaml
Modify: src/main.rs
Create: src/cli.rs
Create: src/config.rs
Create: src/runner.rs
Create: src/protocols/mod.rs
Create: src/protocols/openai.rs
Create: src/protocols/anthropic.rs
Create: src/benchmarks/mod.rs
Create: src/benchmarks/aime.rs
Create: src/benchmarks/gpqa.rs
Create: src/benchmarks/judge.rs
Create: src/metrics.rs
Set Cargo.toml dependencies to:
[package]
name = "lq_token_test"
version = "0.1.0"
edition = "2024"
[dependencies]
anyhow = "1"
chrono = { version = "0.4", features = ["serde"] }
clap = { version = "4", features = ["derive"] }
futures = "0.3"
hdrhistogram = "7"
indicatif = "0.17"
regex = "1"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
sha2 = "0.10"
thiserror = "2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
url = "2"
[dev-dependencies]
tempfile = "3"
wiremock = "0.6"
Set .gitignore to:
/target
/data/benchmarks
/reports
Create config.example.yaml:
default_provider: openai
providers:
openai:
protocol: openai
base_url: "https://relay.example.com/v1"
api_token: "${OPENAI_RELAY_TOKEN}"
default_model: "gpt-4o-mini"
anthropic:
protocol: anthropic
base_url: "https://relay.example.com"
api_token: "${ANTHROPIC_RELAY_TOKEN}"
default_model: "claude-3-5-sonnet-latest"
benchmarks:
data_dir: "data/benchmarks"
aime2026:
source: "huggingface:MathArena/aime_2026"
split: "train"
gpqa_diamond:
source: "huggingface:Idavidrein/gpqa"
split: "gpqa_diamond"
Replace src/main.rs with:
mod benchmarks;
mod cli;
mod config;
mod metrics;
mod protocols;
mod runner;
use anyhow::Result;
use clap::Parser;
use cli::Cli;
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
let cli = Cli::parse();
cli::dispatch(cli).await
}
Create placeholder modules that compile:
// src/cli.rs
use anyhow::Result;
use clap::{Parser, Subcommand};
use std::path::PathBuf;
#[derive(Debug, Parser)]
#[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")]
pub struct Cli {
#[command(subcommand)]
pub command: Command,
}
#[derive(Debug, Subcommand)]
pub enum Command {
Check {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long)]
prompt: String,
},
Dataset {
#[command(subcommand)]
command: DatasetCommand,
},
Bench {
#[command(subcommand)]
command: BenchCommand,
},
Rpm {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long)]
rpm: u32,
#[arg(long)]
duration: String,
#[arg(long)]
prompt: String,
},
}
#[derive(Debug, Subcommand)]
pub enum DatasetCommand {
Fetch { dataset: String },
}
#[derive(Debug, Subcommand)]
pub enum BenchCommand {
Aime2026 {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long, default_value_t = 4)]
concurrency: usize,
#[arg(long)]
limit: Option<usize>,
},
GpqaDiamond {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long, default_value_t = 4)]
concurrency: usize,
#[arg(long)]
limit: Option<usize>,
},
}
pub async fn dispatch(cli: Cli) -> Result<()> {
match cli.command {
Command::Check { .. } => anyhow::bail!("check is not implemented yet"),
Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"),
Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"),
Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"),
}
}
Create each other module with one line:
// src/config.rs, src/runner.rs, src/metrics.rs
For directories:
// src/protocols/mod.rs
pub mod anthropic;
pub mod openai;
// src/protocols/openai.rs
// src/protocols/anthropic.rs
// src/benchmarks/mod.rs
pub mod aime;
pub mod gpqa;
pub mod judge;
// src/benchmarks/aime.rs
// src/benchmarks/gpqa.rs
// src/benchmarks/judge.rs
Run: cargo fmt
Run: cargo test
Expected: build succeeds with no tests or only placeholder warnings.
git add Cargo.toml Cargo.lock .gitignore config.example.yaml src
git commit -m "chore: initialize cli project skeleton"
Files:
Modify: src/config.rs
Modify: src/cli.rs
Add tests to src/config.rs:
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
#[test]
fn loads_provider_and_expands_env_token() {
unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") };
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.yaml");
fs::write(
&path,
r#"
default_provider: openai
providers:
openai:
protocol: openai
base_url: "https://example.com/v1"
api_token: "${LQ_TEST_TOKEN}"
default_model: "gpt-test"
benchmarks:
data_dir: "data/benchmarks"
"#,
)
.unwrap();
let config = AppConfig::load(&path).unwrap();
let provider = config.provider(None).unwrap();
assert_eq!(provider.api_token, "secret-token");
assert_eq!(provider.default_model, "gpt-test");
assert_eq!(provider.protocol, ProtocolKind::Openai);
}
#[test]
fn rejects_missing_env_token() {
unsafe { std::env::remove_var("LQ_MISSING_TOKEN") };
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.yaml");
fs::write(
&path,
r#"
default_provider: openai
providers:
openai:
protocol: openai
base_url: "https://example.com/v1"
api_token: "${LQ_MISSING_TOKEN}"
default_model: "gpt-test"
"#,
)
.unwrap();
let err = AppConfig::load(&path).unwrap_err().to_string();
assert!(err.contains("LQ_MISSING_TOKEN"));
}
}
Run: cargo test config::tests
Expected: fails because AppConfig and ProtocolKind are not implemented.
Implement src/config.rs:
use serde::Deserialize;
use std::{collections::BTreeMap, fs, path::Path};
use thiserror::Error;
#[derive(Debug, Error)]
pub enum ConfigError {
#[error("failed to read config {path}: {source}")]
Read {
path: String,
#[source]
source: std::io::Error,
},
#[error("failed to parse yaml config: {0}")]
Parse(#[from] serde_yaml::Error),
#[error("unknown provider '{0}'")]
UnknownProvider(String),
#[error("config has no default_provider and no --provider was supplied")]
MissingProvider,
#[error("environment variable '{0}' referenced by config is not set")]
MissingEnv(String),
}
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum ProtocolKind {
Openai,
Anthropic,
}
#[derive(Debug, Clone, Deserialize)]
pub struct ProviderConfig {
pub protocol: ProtocolKind,
pub base_url: String,
pub api_token: String,
pub default_model: String,
}
#[derive(Debug, Clone, Deserialize, Default)]
pub struct BenchmarkConfig {
#[serde(default = "default_data_dir")]
pub data_dir: String,
#[serde(default)]
pub aime2026: Option<DatasetConfig>,
#[serde(default)]
pub gpqa_diamond: Option<DatasetConfig>,
}
#[derive(Debug, Clone, Deserialize)]
pub struct DatasetConfig {
pub source: String,
pub split: String,
}
#[derive(Debug, Clone, Deserialize)]
pub struct AppConfig {
pub default_provider: Option<String>,
pub providers: BTreeMap<String, ProviderConfig>,
#[serde(default)]
pub benchmarks: BenchmarkConfig,
}
fn default_data_dir() -> String {
"data/benchmarks".to_string()
}
impl AppConfig {
pub fn load(path: &Path) -> Result<Self, ConfigError> {
let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read {
path: path.display().to_string(),
source,
})?;
let expanded = expand_env_refs(&raw)?;
Ok(serde_yaml::from_str(&expanded)?)
}
pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> {
let name = match provider {
Some(name) => name,
None => self
.default_provider
.as_deref()
.ok_or(ConfigError::MissingProvider)?,
};
self.providers
.get(name)
.ok_or_else(|| ConfigError::UnknownProvider(name.to_string()))
}
}
fn expand_env_refs(input: &str) -> Result<String, ConfigError> {
let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex");
let mut output = String::with_capacity(input.len());
let mut last = 0;
for caps in re.captures_iter(input) {
let whole = caps.get(0).expect("whole match");
let name = caps.get(1).expect("env name").as_str();
output.push_str(&input[last..whole.start()]);
let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?;
output.push_str(&value);
last = whole.end();
}
output.push_str(&input[last..]);
Ok(output)
}
Run: cargo test config::tests
Expected: both config tests pass.
git add src/config.rs src/cli.rs
git commit -m "feat: load yaml relay config"
Files:
Modify: src/cli.rs
Modify: src/benchmarks/mod.rs
Modify: src/benchmarks/aime.rs
Modify: src/benchmarks/gpqa.rs
Modify: src/benchmarks/judge.rs
Add tests to src/benchmarks/judge.rs:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_final_integer() {
assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string()));
assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string()));
}
#[test]
fn extracts_choice_letter() {
assert_eq!(extract_choice("Answer: C"), Some('C'));
assert_eq!(extract_choice("I choose (b)."), Some('B'));
}
}
Implement src/benchmarks/judge.rs:
pub fn extract_final_integer(text: &str) -> Option<String> {
let re = regex::Regex::new(r"-?\d+").expect("valid integer regex");
re.find_iter(text).last().map(|m| m.as_str().to_string())
}
pub fn extract_choice(text: &str) -> Option<char> {
let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex");
re.captures_iter(text)
.last()
.and_then(|caps| caps.get(1))
.and_then(|m| m.as_str().chars().next())
.map(|c| c.to_ascii_uppercase())
}
pub fn judge_integer(output: &str, expected: &str) -> bool {
extract_final_integer(output).as_deref() == Some(expected.trim())
}
pub fn judge_choice(output: &str, expected: char) -> bool {
extract_choice(output) == Some(expected.to_ascii_uppercase())
}
Add tests in src/benchmarks/aime.rs and src/benchmarks/gpqa.rs using inline rows:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn builds_aime_prompt() {
let case = AimeCase {
id: "1".to_string(),
problem: "What is 20 + 22?".to_string(),
answer: "42".to_string(),
};
assert!(case.prompt().contains("What is 20 + 22?"));
assert!(case.prompt().contains("final integer answer"));
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn builds_gpqa_prompt() {
let case = GpqaCase {
id: "gpqa_1".to_string(),
question: "Which option is correct?".to_string(),
choices: [
("A".to_string(), "Alpha".to_string()),
("B".to_string(), "Beta".to_string()),
("C".to_string(), "Gamma".to_string()),
("D".to_string(), "Delta".to_string()),
],
answer: 'B',
};
let prompt = case.prompt();
assert!(prompt.contains("A. Alpha"));
assert!(prompt.contains("answer with exactly one letter"));
}
}
Implement enough for prompts and later download wiring:
// src/benchmarks/aime.rs
#[derive(Debug, Clone)]
pub struct AimeCase {
pub id: String,
pub problem: String,
pub answer: String,
}
impl AimeCase {
pub fn prompt(&self) -> String {
format!(
"Solve the following AIME problem. Return only the final integer answer.\n\n{}",
self.problem
)
}
}
// src/benchmarks/gpqa.rs
#[derive(Debug, Clone)]
pub struct GpqaCase {
pub id: String,
pub question: String,
pub choices: [(String, String); 4],
pub answer: char,
}
impl GpqaCase {
pub fn prompt(&self) -> String {
format!(
"Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}",
self.question,
self.choices[0].0,
self.choices[0].1,
self.choices[1].0,
self.choices[1].1,
self.choices[2].0,
self.choices[2].1,
self.choices[3].0,
self.choices[3].1
)
}
}
Add download functions that use Hugging Face raw URLs:
// src/benchmarks/mod.rs
use anyhow::{Context, Result};
use chrono::Utc;
use sha2::{Digest, Sha256};
use std::{fs, path::{Path, PathBuf}};
pub mod aime;
pub mod gpqa;
pub mod judge;
pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> {
match dataset {
"aime2026" => fetch_to_dir(
"aime2026",
"https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet",
data_dir,
None,
)
.await,
"gpqa-diamond" => fetch_to_dir(
"gpqa_diamond",
"https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv",
data_dir,
std::env::var("HF_TOKEN").ok(),
)
.await,
other => anyhow::bail!("unknown dataset '{other}'"),
}
}
async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> {
let dir = data_dir.join(name);
fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?;
let filename = url.rsplit('/').next().unwrap_or("dataset");
let output = dir.join(filename);
let client = reqwest::Client::new();
let mut request = client.get(url);
if let Some(token) = token {
request = request.bearer_auth(token);
}
let bytes = request.send().await?.error_for_status()?.bytes().await?;
fs::write(&output, &bytes)?;
let hash = Sha256::digest(&bytes);
let metadata = format!(
"name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n",
Utc::now().to_rfc3339(),
hash,
bytes.len()
);
fs::write(dir.join("metadata.yaml"), metadata)?;
Ok(output)
}
Update src/cli.rs dispatch for DatasetCommand::Fetch to load default data dir from config when config.yaml exists, otherwise use data/benchmarks.
Run: cargo test benchmarks
Expected: judge and prompt tests pass. Network fetch is not tested by default.
git add src/benchmarks src/cli.rs
git commit -m "feat: add benchmark dataset fetch and prompts"
Files:
Modify: src/runner.rs
Modify: src/protocols/mod.rs
Modify: src/protocols/openai.rs
Modify: src/protocols/anthropic.rs
Modify: src/cli.rs
Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text.
Implement provider-neutral types:
#[derive(Debug, Clone)]
pub struct ModelRequest {
pub base_url: String,
pub api_token: String,
pub model: String,
pub prompt: String,
pub temperature: f32,
pub max_tokens: u32,
}
#[derive(Debug, Clone)]
pub struct ModelResponse {
pub text: String,
pub status: u16,
pub elapsed_ms: u128,
}
Post to {base_url}/chat/completions with Authorization: Bearer <token> and parse choices[0].message.content.
Post to {base_url}/v1/messages with x-api-key, anthropic-version: 2023-06-01, and parse the first text block in content.
Load config, resolve provider and model, run one request, then print status, elapsed time, and response text.
Run: cargo test protocols runner
Expected: mocked protocol tests pass.
git add src/runner.rs src/protocols src/cli.rs
git commit -m "feat: add openai and anthropic request adapters"
Files:
Modify: src/metrics.rs
Modify: src/benchmarks/mod.rs
Modify: src/benchmarks/aime.rs
Modify: src/benchmarks/gpqa.rs
Modify: src/cli.rs
Test success/error counts, accuracy, and latency percentile calculation.
Use hdrhistogram::Histogram<u64> for elapsed milliseconds and counters for success, failure, correct, and total judged.
For AIME 2026 and GPQA-Diamond, read local files under data/benchmarks. If data is missing, return a clear error: missing local dataset; run lq_token_test dataset fetch <name>.
Use futures::stream with buffer_unordered(concurrency) to run cases concurrently. Apply limit before execution. Judge each response and update metrics.
Parse duration strings like 60s and 5m, calculate delay between requests from rpm, run repeated requests, and print latency/error summary.
Run: cargo test
Expected: unit tests pass.
git add src/metrics.rs src/benchmarks src/cli.rs
git commit -m "feat: run benchmark and rpm tests"
Files:
Create: README.md
Modify: config.example.yaml
Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test.
Run:
cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
Expected: all pass.
Run:
cargo run -- --help
cargo run -- dataset fetch aime2026
Expected: help renders; AIME fetch creates data/benchmarks/aime2026/metadata.yaml.
git add README.md config.example.yaml
git commit -m "docs: add usage guide"
aime2026 and gpqa-diamond in CLI; local directory names are aime2026 and gpqa_diamond; module names are aime and gpqa.