| @@ -0,0 +1,155 @@ | |||||
| # lq_token_test | |||||
| `lq_token_test` is a Rust CLI for checking LLM relay compatibility, running small benchmark probes, and testing request pacing against RPM targets. It supports OpenAI-compatible and Anthropic-compatible relay protocols, fetches local benchmark datasets, prints terminal summaries, and writes JSON reports for later comparison. | |||||
| ## Build And Test | |||||
| ```bash | |||||
| cargo fmt --check | |||||
| cargo test | |||||
| cargo clippy --all-targets -- -D warnings | |||||
| cargo run -- --help | |||||
| ``` | |||||
| Build a release binary with: | |||||
| ```bash | |||||
| cargo build --release | |||||
| ``` | |||||
| ## Configuration | |||||
| Copy `config.example.yaml` to `config.yaml`, then set relay URLs, model names, and token environment variables for your environment. | |||||
| ```yaml | |||||
| default_provider: openai | |||||
| providers: | |||||
| openai: | |||||
| protocol: openai | |||||
| base_url: "https://relay.example.com/v1" | |||||
| api_token: "${OPENAI_RELAY_TOKEN}" | |||||
| default_model: "gpt-4o-mini" | |||||
| anthropic: | |||||
| protocol: anthropic | |||||
| base_url: "https://relay.example.com" | |||||
| api_token: "${ANTHROPIC_RELAY_TOKEN}" | |||||
| default_model: "claude-3-5-sonnet-latest" | |||||
| benchmarks: | |||||
| data_dir: "data/benchmarks" | |||||
| aime2026: | |||||
| source: "huggingface:MathArena/aime_2026" | |||||
| split: "train" | |||||
| gpqa_diamond: | |||||
| source: "huggingface:Idavidrein/gpqa" | |||||
| split: "gpqa_diamond" | |||||
| ``` | |||||
| Values written as `${ENV_NAME}` are expanded when the config is loaded. For example: | |||||
| ```bash | |||||
| export OPENAI_RELAY_TOKEN="..." | |||||
| export ANTHROPIC_RELAY_TOKEN="..." | |||||
| ``` | |||||
| ## Dataset Fetching | |||||
| Fetch AIME 2026 into the configured benchmark data directory: | |||||
| ```bash | |||||
| cargo run -- dataset fetch aime2026 | |||||
| ``` | |||||
| The current AIME fetch path reads Hugging Face rows and normalizes them to local JSONL at `data/benchmarks/aime2026/aime2026.jsonl`, with `id`, `problem`, and `answer` fields. Dataset metadata is written next to the downloaded file. | |||||
| Fetch GPQA-Diamond: | |||||
| ```bash | |||||
| cargo run -- dataset fetch gpqa-diamond | |||||
| ``` | |||||
| GPQA-Diamond is downloaded as `data/benchmarks/gpqa_diamond/gpqa_diamond.csv`. If Hugging Face requires authentication, set `HF_TOKEN` before fetching: | |||||
| ```bash | |||||
| export HF_TOKEN="..." | |||||
| ``` | |||||
| Dataset files under `data/benchmarks` are ignored by git. | |||||
| ## Relay Checks | |||||
| Run a simple OpenAI-compatible relay check: | |||||
| ```bash | |||||
| cargo run -- check \ | |||||
| --config config.yaml \ | |||||
| --provider openai \ | |||||
| --model gpt-4o-mini \ | |||||
| --prompt "Reply with the word ready." | |||||
| ``` | |||||
| Run a simple Anthropic-compatible relay check: | |||||
| ```bash | |||||
| cargo run -- check \ | |||||
| --config config.yaml \ | |||||
| --provider anthropic \ | |||||
| --model claude-3-5-sonnet-latest \ | |||||
| --prompt "Reply with the word ready." | |||||
| ``` | |||||
| The check command prints the HTTP status, elapsed milliseconds, and model text. | |||||
| ## Benchmarks | |||||
| Run an AIME 2026 benchmark: | |||||
| ```bash | |||||
| cargo run -- bench aime2026 \ | |||||
| --config config.yaml \ | |||||
| --provider openai \ | |||||
| --model gpt-4o-mini \ | |||||
| --concurrency 4 \ | |||||
| --limit 10 | |||||
| ``` | |||||
| Run a GPQA-Diamond benchmark: | |||||
| ```bash | |||||
| cargo run -- bench gpqa-diamond \ | |||||
| --config config.yaml \ | |||||
| --provider openai \ | |||||
| --model gpt-4o-mini \ | |||||
| --concurrency 4 \ | |||||
| --limit 10 | |||||
| ``` | |||||
| Omit `--limit` to run all locally available cases. | |||||
| ## RPM Testing | |||||
| Run an RPM test: | |||||
| ```bash | |||||
| cargo run -- rpm \ | |||||
| --config config.yaml \ | |||||
| --provider openai \ | |||||
| --model gpt-4o-mini \ | |||||
| --rpm 60 \ | |||||
| --duration 60s \ | |||||
| --prompt "Reply with pong." | |||||
| ``` | |||||
| Durations use `s` or `m`, such as `30s` or `5m`. | |||||
| ## Reports | |||||
| Benchmark and RPM commands print a terminal summary with success counts, failures, latency percentiles, errors, and the report path. JSON reports are written under `reports/*.json`; the `reports` directory is ignored by git. | |||||
| Benchmark reports include `wrong_cases`, with each wrong case containing the case id, question, expected answer, extracted actual answer, and raw model output. RPM reports include request counts, target RPM, latency, and error counts. | |||||
| ## Comparing Scores | |||||
| Use these results as relay benchmark signals, not absolute proof by themselves. To compare against official scores or another run, align the same dataset and source, prompt text, temperature, `max_tokens`, sample limit, and scoring logic. Differences in any of those inputs can make the reported accuracy diverge from official numbers or other benchmark harnesses. | |||||