# lq_token_test Design

Date: 2026-05-06

## Goal

Build a Rust CLI tool for testing an LLM relay service. The tool should start with reliable single-request checks, then grow into RPM/concurrency testing and dataset-based accuracy evaluation.

The first implementation will use a modular single-crate architecture. This keeps initialization light while still giving clear boundaries for config loading, CLI parsing, protocol adapters, request execution, benchmarks, and metrics.

## Scope

First phase:

- Read relay configuration from YAML.
- Manage relay base URLs, API tokens, default models, and provider settings.
- Support OpenAI-compatible chat requests.
- Add Anthropic-compatible request structure behind the same internal runner boundary.
- Provide CLI subcommands for single checks, official benchmark tests, and RPM/concurrency tests.
- Support the official GSM8K test split as the first benchmark dataset.
- Include GSM8K-style numeric answer judging.
- Report success count, error count, latency summaries, and benchmark accuracy.

Later phase:

- Add stricter benchmark profiles for comparing against official model reports.
- Record prompt template, dataset version, sampling parameters, model identity, and scoring method.
- Add more official benchmarks such as MMLU-style multiple-choice tasks after GSM8K is stable.
- Add custom local JSONL ingestion as an extension, not as the first benchmark path.

## Architecture

Use scheme B: one binary crate split into internal modules.

Planned structure:

```text
src/
  main.rs
  cli.rs
  config.rs
  runner.rs
  metrics.rs
  protocols/
    mod.rs
    openai.rs
    anthropic.rs
  benchmarks/
    mod.rs
    gsm8k.rs
    judge.rs
```

Module responsibilities:

- `cli`: Defines subcommands and flags using `clap`.
- `config`: Loads YAML config, resolves environment variable references, and validates provider settings.
- `protocols`: Converts internal request data into provider-specific HTTP requests and parses responses.
- `runner`: Executes one logical model request and returns response text, elapsed time, provider metadata, and errors.
- `benchmarks`: Loads official benchmark datasets, runs them through the runner, and judges answers.
- `metrics`: Aggregates latency, success rate, error distribution, and accuracy summaries.
- `main`: Wires the CLI, config, runner, and command handlers together.

## CLI Shape

Initial commands:

```bash
lq_token_test check --config config.yaml --provider openai --model gpt-4o-mini --prompt "hello"
lq_token_test bench gsm8k --config config.yaml --provider openai --model gpt-4o-mini --limit 100 --concurrency 4
lq_token_test bench gsm8k --config config.yaml --provider anthropic --model claude-3-5-sonnet-latest --limit 100 --concurrency 4
lq_token_test rpm --config config.yaml --provider openai --rpm 60 --duration 60s --prompt "hello"
```

The `check` command proves that a relay, token, model, and protocol shape work.

The `bench` command runs an official benchmark dataset and reports accuracy plus request metrics.

The `rpm` command sends repeated requests at a target rate and reports latency and error behavior.

## Config Format

Example:

```yaml
default_provider: openai

providers:
  openai:
    protocol: openai
    base_url: "https://relay.example.com/v1"
    api_token: "${OPENAI_RELAY_TOKEN}"
    default_model: "gpt-4o-mini"

  anthropic:
    protocol: anthropic
    base_url: "https://relay.example.com"
    api_token: "${ANTHROPIC_RELAY_TOKEN}"
    default_model: "claude-3-5-sonnet-latest"

benchmarks:
  cache_dir: ".cache/lq_token_test/benchmarks"
  gsm8k:
    source: "official_openai_github"
    split: "test"
```

API tokens should be allowed directly in YAML for local testing, but environment variable references are preferred.

## Benchmark Data

First benchmark dataset:

- GSM8K official test split from OpenAI's `grade-school-math` dataset.
- Source repository: `openai/grade-school-math`.
- Source file: `grade_school_math/data/test.jsonl`.
- Expected format: each line contains `question` and `answer`.
- The final answer is extracted from the official `answer` field using the GSM8K `#### <answer>` convention.

The CLI should be able to download and cache the official test file when network access is available. It should also accept a local official GSM8K `test.jsonl` path for offline or pinned-version runs.

Judging rules:

- `gsm8k`: extract the final numeric answer from model output and compare with the expected answer.

The first phase accuracy score is an official-dataset relay evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset commit, prompt template, sample count, temperature, and scoring method are aligned with the official report.

## Dependencies

Recommended crates:

- `clap`: CLI parser and subcommands.
- `serde`, `serde_yaml`, `serde_json`: config and dataset parsing.
- `tokio`: async runtime.
- `reqwest`: HTTP client.
- `anyhow`: simple top-level CLI error handling.
- `thiserror`: structured module-level errors.
- `tracing`, `tracing-subscriber`: logs.
- `indicatif`: progress display for benchmark and RPM runs.
- `hdrhistogram`: latency percentiles.
- `regex`: answer extraction for GSM8K-style judging.

Prefer `reqwest` with Rustls TLS. Avoid provider SDKs in the first phase so protocol compatibility remains transparent and easy to inspect.

## Error Handling

The CLI should surface concise user-facing errors:

- Missing config file.
- Unknown provider.
- Missing API token or unresolved environment variable.
- Unsupported protocol.
- HTTP status failures.
- Provider response parse failures.
- Invalid official benchmark data lines.

Benchmark and RPM commands should continue after per-request failures and include failures in the final summary.

## Testing

Initial tests should cover:

- YAML config loading and environment variable expansion.
- Official GSM8K test data parsing.
- GSM8K-style numeric extraction.
- Metrics aggregation.

Network tests should be kept opt-in because they require real relay credentials.

## First Implementation Decisions

- OpenAI-compatible support starts with `/chat/completions`.
- Anthropic support includes a real adapter boundary and request shape in the first pass.
- The first benchmark target is the official GSM8K test split.
- Benchmark prompts ask the model to solve the problem and end with a single final numeric answer. The exact prompt template is recorded in benchmark output.
- Benchmark runs should record provider, model, dataset source, dataset split, optional dataset commit, limit, concurrency, temperature, max tokens, accuracy, latency percentiles, and error summary.