api模型检测
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

23 KiB

lq_token_test Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond.

Architecture: Keep the project as one binary crate with focused internal modules: cli, config, runner, protocols, benchmarks, and metrics. Datasets are not committed; the CLI downloads them into data/benchmarks/, writes metadata, and benchmark commands read only local files.

Tech Stack: Rust 2024, clap, serde, serde_yaml, serde_json, tokio, reqwest with Rustls, anyhow, thiserror, tracing, indicatif, hdrhistogram, regex, sha2, chrono, futures.


File Structure

  • Modify Cargo.toml: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies.
  • Modify .gitignore: keep target/ and add local benchmark data/output directories.
  • Create config.example.yaml: example relay and benchmark configuration.
  • Modify src/main.rs: async entry point, logging, command dispatch.
  • Create src/cli.rs: clap command definitions.
  • Create src/config.rs: YAML loading, provider lookup, environment expansion.
  • Create src/runner.rs: provider-neutral request and response types plus request execution timing.
  • Create src/protocols/mod.rs: protocol enum and dispatch.
  • Create src/protocols/openai.rs: OpenAI /chat/completions request/response adapter.
  • Create src/protocols/anthropic.rs: Anthropic /v1/messages request/response adapter.
  • Create src/benchmarks/mod.rs: benchmark command orchestration and shared record types.
  • Create src/benchmarks/aime.rs: AIME 2026 download, parse, prompt, judge integration.
  • Create src/benchmarks/gpqa.rs: GPQA-Diamond download, parse, prompt, judge integration.
  • Create src/benchmarks/judge.rs: numeric and multiple-choice answer extraction.
  • Create src/metrics.rs: latency, success/error, and accuracy summaries.

Task 1: Dependencies And Project Skeleton

Files:

  • Modify: Cargo.toml

  • Modify: .gitignore

  • Create: config.example.yaml

  • Modify: src/main.rs

  • Create: src/cli.rs

  • Create: src/config.rs

  • Create: src/runner.rs

  • Create: src/protocols/mod.rs

  • Create: src/protocols/openai.rs

  • Create: src/protocols/anthropic.rs

  • Create: src/benchmarks/mod.rs

  • Create: src/benchmarks/aime.rs

  • Create: src/benchmarks/gpqa.rs

  • Create: src/benchmarks/judge.rs

  • Create: src/metrics.rs

Set Cargo.toml dependencies to:

[package]
name = "lq_token_test"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "1"
chrono = { version = "0.4", features = ["serde"] }
clap = { version = "4", features = ["derive"] }
futures = "0.3"
hdrhistogram = "7"
indicatif = "0.17"
regex = "1"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
sha2 = "0.10"
thiserror = "2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
url = "2"

[dev-dependencies]
tempfile = "3"
wiremock = "0.6"

Set .gitignore to:

/target
/data/benchmarks
/reports

Create config.example.yaml:

default_provider: openai

providers:
  openai:
    protocol: openai
    base_url: "https://relay.example.com/v1"
    api_token: "${OPENAI_RELAY_TOKEN}"
    default_model: "gpt-4o-mini"

  anthropic:
    protocol: anthropic
    base_url: "https://relay.example.com"
    api_token: "${ANTHROPIC_RELAY_TOKEN}"
    default_model: "claude-3-5-sonnet-latest"

benchmarks:
  data_dir: "data/benchmarks"
  aime2026:
    source: "huggingface:MathArena/aime_2026"
    split: "train"
  gpqa_diamond:
    source: "huggingface:Idavidrein/gpqa"
    split: "gpqa_diamond"

Replace src/main.rs with:

mod benchmarks;
mod cli;
mod config;
mod metrics;
mod protocols;
mod runner;

use anyhow::Result;
use clap::Parser;
use cli::Cli;

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
        .init();

    let cli = Cli::parse();
    cli::dispatch(cli).await
}

Create placeholder modules that compile:

// src/cli.rs
use anyhow::Result;
use clap::{Parser, Subcommand};
use std::path::PathBuf;

#[derive(Debug, Parser)]
#[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")]
pub struct Cli {
    #[command(subcommand)]
    pub command: Command,
}

#[derive(Debug, Subcommand)]
pub enum Command {
    Check {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long)]
        prompt: String,
    },
    Dataset {
        #[command(subcommand)]
        command: DatasetCommand,
    },
    Bench {
        #[command(subcommand)]
        command: BenchCommand,
    },
    Rpm {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long)]
        rpm: u32,
        #[arg(long)]
        duration: String,
        #[arg(long)]
        prompt: String,
    },
}

#[derive(Debug, Subcommand)]
pub enum DatasetCommand {
    Fetch { dataset: String },
}

#[derive(Debug, Subcommand)]
pub enum BenchCommand {
    Aime2026 {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long, default_value_t = 4)]
        concurrency: usize,
        #[arg(long)]
        limit: Option<usize>,
    },
    GpqaDiamond {
        #[arg(long, default_value = "config.yaml")]
        config: PathBuf,
        #[arg(long)]
        provider: Option<String>,
        #[arg(long)]
        model: Option<String>,
        #[arg(long, default_value_t = 4)]
        concurrency: usize,
        #[arg(long)]
        limit: Option<usize>,
    },
}

pub async fn dispatch(cli: Cli) -> Result<()> {
    match cli.command {
        Command::Check { .. } => anyhow::bail!("check is not implemented yet"),
        Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"),
        Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"),
        Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"),
    }
}

Create each other module with one line:

// src/config.rs, src/runner.rs, src/metrics.rs

For directories:

// src/protocols/mod.rs
pub mod anthropic;
pub mod openai;
// src/protocols/openai.rs
// src/protocols/anthropic.rs
// src/benchmarks/mod.rs
pub mod aime;
pub mod gpqa;
pub mod judge;
// src/benchmarks/aime.rs
// src/benchmarks/gpqa.rs
// src/benchmarks/judge.rs

Run: cargo fmt

Run: cargo test

Expected: build succeeds with no tests or only placeholder warnings.

git add Cargo.toml Cargo.lock .gitignore config.example.yaml src
git commit -m "chore: initialize cli project skeleton"

Task 2: Config Loading

Files:

  • Modify: src/config.rs

  • Modify: src/cli.rs

Add tests to src/config.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use std::fs;

    #[test]
    fn loads_provider_and_expands_env_token() {
        unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") };
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("config.yaml");
        fs::write(
            &path,
            r#"
default_provider: openai
providers:
  openai:
    protocol: openai
    base_url: "https://example.com/v1"
    api_token: "${LQ_TEST_TOKEN}"
    default_model: "gpt-test"
benchmarks:
  data_dir: "data/benchmarks"
"#,
        )
        .unwrap();

        let config = AppConfig::load(&path).unwrap();
        let provider = config.provider(None).unwrap();
        assert_eq!(provider.api_token, "secret-token");
        assert_eq!(provider.default_model, "gpt-test");
        assert_eq!(provider.protocol, ProtocolKind::Openai);
    }

    #[test]
    fn rejects_missing_env_token() {
        unsafe { std::env::remove_var("LQ_MISSING_TOKEN") };
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("config.yaml");
        fs::write(
            &path,
            r#"
default_provider: openai
providers:
  openai:
    protocol: openai
    base_url: "https://example.com/v1"
    api_token: "${LQ_MISSING_TOKEN}"
    default_model: "gpt-test"
"#,
        )
        .unwrap();

        let err = AppConfig::load(&path).unwrap_err().to_string();
        assert!(err.contains("LQ_MISSING_TOKEN"));
    }
}

Run: cargo test config::tests

Expected: fails because AppConfig and ProtocolKind are not implemented.

Implement src/config.rs:

use serde::Deserialize;
use std::{collections::BTreeMap, fs, path::Path};
use thiserror::Error;

#[derive(Debug, Error)]
pub enum ConfigError {
    #[error("failed to read config {path}: {source}")]
    Read {
        path: String,
        #[source]
        source: std::io::Error,
    },
    #[error("failed to parse yaml config: {0}")]
    Parse(#[from] serde_yaml::Error),
    #[error("unknown provider '{0}'")]
    UnknownProvider(String),
    #[error("config has no default_provider and no --provider was supplied")]
    MissingProvider,
    #[error("environment variable '{0}' referenced by config is not set")]
    MissingEnv(String),
}

#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum ProtocolKind {
    Openai,
    Anthropic,
}

#[derive(Debug, Clone, Deserialize)]
pub struct ProviderConfig {
    pub protocol: ProtocolKind,
    pub base_url: String,
    pub api_token: String,
    pub default_model: String,
}

#[derive(Debug, Clone, Deserialize, Default)]
pub struct BenchmarkConfig {
    #[serde(default = "default_data_dir")]
    pub data_dir: String,
    #[serde(default)]
    pub aime2026: Option<DatasetConfig>,
    #[serde(default)]
    pub gpqa_diamond: Option<DatasetConfig>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct DatasetConfig {
    pub source: String,
    pub split: String,
}

#[derive(Debug, Clone, Deserialize)]
pub struct AppConfig {
    pub default_provider: Option<String>,
    pub providers: BTreeMap<String, ProviderConfig>,
    #[serde(default)]
    pub benchmarks: BenchmarkConfig,
}

fn default_data_dir() -> String {
    "data/benchmarks".to_string()
}

impl AppConfig {
    pub fn load(path: &Path) -> Result<Self, ConfigError> {
        let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read {
            path: path.display().to_string(),
            source,
        })?;
        let expanded = expand_env_refs(&raw)?;
        Ok(serde_yaml::from_str(&expanded)?)
    }

    pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> {
        let name = match provider {
            Some(name) => name,
            None => self
                .default_provider
                .as_deref()
                .ok_or(ConfigError::MissingProvider)?,
        };
        self.providers
            .get(name)
            .ok_or_else(|| ConfigError::UnknownProvider(name.to_string()))
    }
}

fn expand_env_refs(input: &str) -> Result<String, ConfigError> {
    let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex");
    let mut output = String::with_capacity(input.len());
    let mut last = 0;

    for caps in re.captures_iter(input) {
        let whole = caps.get(0).expect("whole match");
        let name = caps.get(1).expect("env name").as_str();
        output.push_str(&input[last..whole.start()]);
        let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?;
        output.push_str(&value);
        last = whole.end();
    }

    output.push_str(&input[last..]);
    Ok(output)
}

Run: cargo test config::tests

Expected: both config tests pass.

git add src/config.rs src/cli.rs
git commit -m "feat: load yaml relay config"

Task 3: Dataset Download And Parsing

Files:

  • Modify: src/cli.rs

  • Modify: src/benchmarks/mod.rs

  • Modify: src/benchmarks/aime.rs

  • Modify: src/benchmarks/gpqa.rs

  • Modify: src/benchmarks/judge.rs

Add tests to src/benchmarks/judge.rs:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn extracts_final_integer() {
        assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string()));
        assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string()));
    }

    #[test]
    fn extracts_choice_letter() {
        assert_eq!(extract_choice("Answer: C"), Some('C'));
        assert_eq!(extract_choice("I choose (b)."), Some('B'));
    }
}

Implement src/benchmarks/judge.rs:

pub fn extract_final_integer(text: &str) -> Option<String> {
    let re = regex::Regex::new(r"-?\d+").expect("valid integer regex");
    re.find_iter(text).last().map(|m| m.as_str().to_string())
}

pub fn extract_choice(text: &str) -> Option<char> {
    let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex");
    re.captures_iter(text)
        .last()
        .and_then(|caps| caps.get(1))
        .and_then(|m| m.as_str().chars().next())
        .map(|c| c.to_ascii_uppercase())
}

pub fn judge_integer(output: &str, expected: &str) -> bool {
    extract_final_integer(output).as_deref() == Some(expected.trim())
}

pub fn judge_choice(output: &str, expected: char) -> bool {
    extract_choice(output) == Some(expected.to_ascii_uppercase())
}

Add tests in src/benchmarks/aime.rs and src/benchmarks/gpqa.rs using inline rows:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn builds_aime_prompt() {
        let case = AimeCase {
            id: "1".to_string(),
            problem: "What is 20 + 22?".to_string(),
            answer: "42".to_string(),
        };
        assert!(case.prompt().contains("What is 20 + 22?"));
        assert!(case.prompt().contains("final integer answer"));
    }
}
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn builds_gpqa_prompt() {
        let case = GpqaCase {
            id: "gpqa_1".to_string(),
            question: "Which option is correct?".to_string(),
            choices: [
                ("A".to_string(), "Alpha".to_string()),
                ("B".to_string(), "Beta".to_string()),
                ("C".to_string(), "Gamma".to_string()),
                ("D".to_string(), "Delta".to_string()),
            ],
            answer: 'B',
        };
        let prompt = case.prompt();
        assert!(prompt.contains("A. Alpha"));
        assert!(prompt.contains("answer with exactly one letter"));
    }
}

Implement enough for prompts and later download wiring:

// src/benchmarks/aime.rs
#[derive(Debug, Clone)]
pub struct AimeCase {
    pub id: String,
    pub problem: String,
    pub answer: String,
}

impl AimeCase {
    pub fn prompt(&self) -> String {
        format!(
            "Solve the following AIME problem. Return only the final integer answer.\n\n{}",
            self.problem
        )
    }
}
// src/benchmarks/gpqa.rs
#[derive(Debug, Clone)]
pub struct GpqaCase {
    pub id: String,
    pub question: String,
    pub choices: [(String, String); 4],
    pub answer: char,
}

impl GpqaCase {
    pub fn prompt(&self) -> String {
        format!(
            "Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}",
            self.question,
            self.choices[0].0,
            self.choices[0].1,
            self.choices[1].0,
            self.choices[1].1,
            self.choices[2].0,
            self.choices[2].1,
            self.choices[3].0,
            self.choices[3].1
        )
    }
}

Add download functions that use Hugging Face raw URLs:

// src/benchmarks/mod.rs
use anyhow::{Context, Result};
use chrono::Utc;
use sha2::{Digest, Sha256};
use std::{fs, path::{Path, PathBuf}};

pub mod aime;
pub mod gpqa;
pub mod judge;

pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> {
    match dataset {
        "aime2026" => fetch_to_dir(
            "aime2026",
            "https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet",
            data_dir,
            None,
        )
        .await,
        "gpqa-diamond" => fetch_to_dir(
            "gpqa_diamond",
            "https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv",
            data_dir,
            std::env::var("HF_TOKEN").ok(),
        )
        .await,
        other => anyhow::bail!("unknown dataset '{other}'"),
    }
}

async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> {
    let dir = data_dir.join(name);
    fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?;
    let filename = url.rsplit('/').next().unwrap_or("dataset");
    let output = dir.join(filename);

    let client = reqwest::Client::new();
    let mut request = client.get(url);
    if let Some(token) = token {
        request = request.bearer_auth(token);
    }
    let bytes = request.send().await?.error_for_status()?.bytes().await?;
    fs::write(&output, &bytes)?;

    let hash = Sha256::digest(&bytes);
    let metadata = format!(
        "name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n",
        Utc::now().to_rfc3339(),
        hash,
        bytes.len()
    );
    fs::write(dir.join("metadata.yaml"), metadata)?;
    Ok(output)
}

Update src/cli.rs dispatch for DatasetCommand::Fetch to load default data dir from config when config.yaml exists, otherwise use data/benchmarks.

Run: cargo test benchmarks

Expected: judge and prompt tests pass. Network fetch is not tested by default.

git add src/benchmarks src/cli.rs
git commit -m "feat: add benchmark dataset fetch and prompts"

Task 4: Protocol Adapters And Single Check

Files:

  • Modify: src/runner.rs

  • Modify: src/protocols/mod.rs

  • Modify: src/protocols/openai.rs

  • Modify: src/protocols/anthropic.rs

  • Modify: src/cli.rs

Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text.

Implement provider-neutral types:

#[derive(Debug, Clone)]
pub struct ModelRequest {
    pub base_url: String,
    pub api_token: String,
    pub model: String,
    pub prompt: String,
    pub temperature: f32,
    pub max_tokens: u32,
}

#[derive(Debug, Clone)]
pub struct ModelResponse {
    pub text: String,
    pub status: u16,
    pub elapsed_ms: u128,
}

Post to {base_url}/chat/completions with Authorization: Bearer <token> and parse choices[0].message.content.

Post to {base_url}/v1/messages with x-api-key, anthropic-version: 2023-06-01, and parse the first text block in content.

Load config, resolve provider and model, run one request, then print status, elapsed time, and response text.

Run: cargo test protocols runner

Expected: mocked protocol tests pass.

git add src/runner.rs src/protocols src/cli.rs
git commit -m "feat: add openai and anthropic request adapters"

Task 5: Metrics, Benchmark Runs, And RPM

Files:

  • Modify: src/metrics.rs

  • Modify: src/benchmarks/mod.rs

  • Modify: src/benchmarks/aime.rs

  • Modify: src/benchmarks/gpqa.rs

  • Modify: src/cli.rs

Test success/error counts, accuracy, and latency percentile calculation.

Use hdrhistogram::Histogram<u64> for elapsed milliseconds and counters for success, failure, correct, and total judged.

For AIME 2026 and GPQA-Diamond, read local files under data/benchmarks. If data is missing, return a clear error: missing local dataset; run lq_token_test dataset fetch <name>.

Use futures::stream with buffer_unordered(concurrency) to run cases concurrently. Apply limit before execution. Judge each response and update metrics.

Parse duration strings like 60s and 5m, calculate delay between requests from rpm, run repeated requests, and print latency/error summary.

Run: cargo test

Expected: unit tests pass.

git add src/metrics.rs src/benchmarks src/cli.rs
git commit -m "feat: run benchmark and rpm tests"

Task 6: Verification And Docs

Files:

  • Create: README.md

  • Modify: config.example.yaml

Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test.

Run:

cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings

Expected: all pass.

Run:

cargo run -- --help
cargo run -- dataset fetch aime2026

Expected: help renders; AIME fetch creates data/benchmarks/aime2026/metadata.yaml.

git add README.md config.example.yaml
git commit -m "docs: add usage guide"

Self-Review

  • Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage.
  • Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior.
  • Type consistency: dataset names are aime2026 and gpqa-diamond in CLI; local directory names are aime2026 and gpqa_diamond; module names are aime and gpqa.