new-api-check/docs/superpowers/specs/2026-05-06-lq-token-test-design.md

lq_token_test Design

Date: 2026-05-06

Goal

Build a Rust CLI tool for testing an LLM relay service. The tool should start with reliable single-request checks, then grow into RPM/concurrency testing and dataset-based accuracy evaluation.

The first implementation will use a modular single-crate architecture. This keeps initialization light while still giving clear boundaries for config loading, CLI parsing, protocol adapters, request execution, benchmarks, and metrics.

Scope

First phase:

• Read relay configuration from YAML.
• Manage relay base URLs, API tokens, default models, and provider settings.
• Support OpenAI-compatible chat requests.
• Add Anthropic-compatible request structure behind the same internal runner boundary.
• Provide CLI subcommands for single checks, official benchmark tests, and RPM/concurrency tests.
• Support AIME 2026 and GPQA-Diamond as the first benchmark datasets.
• Provide dataset download commands so users can fetch official datasets into local storage before running benchmarks.
• Keep downloaded datasets out of git.
• Include numeric answer judging for AIME and multiple-choice judging for GPQA-Diamond.
• Report success count, error count, latency summaries, and benchmark accuracy.

Later phase:

• Add stricter benchmark profiles for comparing against official model reports.
• Record prompt template, dataset version, sampling parameters, model identity, and scoring method.
• Add more official benchmarks such as HLE text-only or HMMT after AIME 2026 and GPQA-Diamond are stable.
• Add custom local JSONL ingestion as an extension, not as the first benchmark path.

Architecture

Use scheme B: one binary crate split into internal modules.

Planned structure:

src/
  main.rs
  cli.rs
  config.rs
  runner.rs
  metrics.rs
  protocols/
    mod.rs
    openai.rs
    anthropic.rs
  benchmarks/
    mod.rs
    aime.rs
    gpqa.rs
    judge.rs

Module responsibilities:

• cli: Defines subcommands and flags using clap.
• config: Loads YAML config, resolves environment variable references, and validates provider settings.
• protocols: Converts internal request data into provider-specific HTTP requests and parses responses.
• runner: Executes one logical model request and returns response text, elapsed time, provider metadata, and errors.
• benchmarks: Loads official benchmark datasets, runs them through the runner, and judges answers.
• metrics: Aggregates latency, success rate, error distribution, and accuracy summaries.
• main: Wires the CLI, config, runner, and command handlers together.

CLI Shape

Initial commands:

lq_token_test check --config config.yaml --provider openai --model gpt-4o-mini --prompt "hello"
lq_token_test dataset fetch aime2026
lq_token_test dataset fetch gpqa-diamond
lq_token_test bench aime2026 --config config.yaml --provider openai --model gpt-4o-mini --concurrency 4
lq_token_test bench gpqa-diamond --config config.yaml --provider anthropic --model claude-3-5-sonnet-latest --limit 100 --concurrency 4
lq_token_test rpm --config config.yaml --provider openai --rpm 60 --duration 60s --prompt "hello"

The check command proves that a relay, token, model, and protocol shape work.

The dataset fetch command downloads an official dataset into local storage and writes source metadata.

The bench command runs an official benchmark dataset and reports accuracy plus request metrics.

The rpm command sends repeated requests at a target rate and reports latency and error behavior.

Config Format

Example:

default_provider: openai

providers:
  openai:
    protocol: openai
    base_url: "https://relay.example.com/v1"
    api_token: "${OPENAI_RELAY_TOKEN}"
    default_model: "gpt-4o-mini"

  anthropic:
    protocol: anthropic
    base_url: "https://relay.example.com"
    api_token: "${ANTHROPIC_RELAY_TOKEN}"
    default_model: "claude-3-5-sonnet-latest"

benchmarks:
  data_dir: "data/benchmarks"
  aime2026:
    source: "huggingface:MathArena/aime_2026"
    split: "train"
  gpqa_diamond:
    source: "huggingface:Idavidrein/gpqa"
    split: "gpqa_diamond"

API tokens should be allowed directly in YAML for local testing, but environment variable references are preferred.

Benchmark Data

First benchmark datasets:

• AIME 2026 from Hugging Face dataset MathArena/aime_2026.
  • Format: parquet.
  • Size: 30 rows.
  • Fields: problem_idx, problem, answer.
  • License: CC BY-NC-SA 4.0.
  • Purpose: current math reasoning benchmark with simple final numeric judging.
• GPQA-Diamond from Hugging Face dataset Idavidrein/gpqa.
  • Format: CSV.
  • Access: requires accepting the dataset conditions on Hugging Face; downloads should support HF_TOKEN.
  • Purpose: current science reasoning benchmark with multiple-choice judging.

Downloaded data should live under data/benchmarks/ by default and should not be committed to git. Each download writes a metadata.yaml with dataset name, source, split, download time, file hash, row count, and license note. Benchmark commands should read from local files and tell the user to run dataset fetch if required data is missing.

Judging rules:

• aime2026: extract the final integer answer from model output and compare with the official answer.
• gpqa-diamond: ask the model to answer with one option letter and compare against the official correct option.

The first phase accuracy score is an official-dataset relay evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset revision, prompt template, sample count, temperature, and scoring method are aligned with the official report.

Dependencies

Recommended crates:

• clap: CLI parser and subcommands.
• serde, serde_yaml, serde_json: config and dataset parsing.
• tokio: async runtime.
• reqwest: HTTP client.
• anyhow: simple top-level CLI error handling.
• thiserror: structured module-level errors.
• tracing, tracing-subscriber: logs.
• indicatif: progress display for benchmark and RPM runs.
• hdrhistogram: latency percentiles.
• regex: answer extraction for numeric and multiple-choice judging.

Prefer reqwest with Rustls TLS. Avoid provider SDKs in the first phase so protocol compatibility remains transparent and easy to inspect.

Error Handling

The CLI should surface concise user-facing errors:

• Missing config file.
• Unknown provider.
• Missing API token or unresolved environment variable.
• Unsupported protocol.
• HTTP status failures.
• Provider response parse failures.
• Invalid official benchmark data lines.
• Missing local benchmark data.
• Dataset download failures.

Benchmark and RPM commands should continue after per-request failures and include failures in the final summary.

Testing

Initial tests should cover:

• YAML config loading and environment variable expansion.
• AIME 2026 data parsing.
• GPQA-Diamond data parsing.
• AIME-style numeric extraction.
• GPQA multiple-choice extraction.
• Metrics aggregation.

Network tests should be kept opt-in because they require real relay credentials.

First Implementation Decisions

• OpenAI-compatible support starts with /chat/completions.
• Anthropic support includes a real adapter boundary and request shape in the first pass.
• The first benchmark targets are AIME 2026 and GPQA-Diamond.
• Datasets are downloaded by command into local storage and are not committed to git.
• Benchmark prompts ask the model to use the answer format required by the dataset. The exact prompt template is recorded in benchmark output.
• Benchmark runs should record provider, model, dataset source, dataset split, optional dataset revision, limit, concurrency, temperature, max tokens, accuracy, latency percentiles, and error summary.