diff --git a/README.md b/README.md new file mode 100644 index 0000000..2e4215f --- /dev/null +++ b/README.md @@ -0,0 +1,155 @@ +# lq_token_test + +`lq_token_test` is a Rust CLI for checking LLM relay compatibility, running small benchmark probes, and testing request pacing against RPM targets. It supports OpenAI-compatible and Anthropic-compatible relay protocols, fetches local benchmark datasets, prints terminal summaries, and writes JSON reports for later comparison. + +## Build And Test + +```bash +cargo fmt --check +cargo test +cargo clippy --all-targets -- -D warnings +cargo run -- --help +``` + +Build a release binary with: + +```bash +cargo build --release +``` + +## Configuration + +Copy `config.example.yaml` to `config.yaml`, then set relay URLs, model names, and token environment variables for your environment. + +```yaml +default_provider: openai + +providers: + openai: + protocol: openai + base_url: "https://relay.example.com/v1" + api_token: "${OPENAI_RELAY_TOKEN}" + default_model: "gpt-4o-mini" + + anthropic: + protocol: anthropic + base_url: "https://relay.example.com" + api_token: "${ANTHROPIC_RELAY_TOKEN}" + default_model: "claude-3-5-sonnet-latest" + +benchmarks: + data_dir: "data/benchmarks" + aime2026: + source: "huggingface:MathArena/aime_2026" + split: "train" + gpqa_diamond: + source: "huggingface:Idavidrein/gpqa" + split: "gpqa_diamond" +``` + +Values written as `${ENV_NAME}` are expanded when the config is loaded. For example: + +```bash +export OPENAI_RELAY_TOKEN="..." +export ANTHROPIC_RELAY_TOKEN="..." +``` + +## Dataset Fetching + +Fetch AIME 2026 into the configured benchmark data directory: + +```bash +cargo run -- dataset fetch aime2026 +``` + +The current AIME fetch path reads Hugging Face rows and normalizes them to local JSONL at `data/benchmarks/aime2026/aime2026.jsonl`, with `id`, `problem`, and `answer` fields. Dataset metadata is written next to the downloaded file. + +Fetch GPQA-Diamond: + +```bash +cargo run -- dataset fetch gpqa-diamond +``` + +GPQA-Diamond is downloaded as `data/benchmarks/gpqa_diamond/gpqa_diamond.csv`. If Hugging Face requires authentication, set `HF_TOKEN` before fetching: + +```bash +export HF_TOKEN="..." +``` + +Dataset files under `data/benchmarks` are ignored by git. + +## Relay Checks + +Run a simple OpenAI-compatible relay check: + +```bash +cargo run -- check \ + --config config.yaml \ + --provider openai \ + --model gpt-4o-mini \ + --prompt "Reply with the word ready." +``` + +Run a simple Anthropic-compatible relay check: + +```bash +cargo run -- check \ + --config config.yaml \ + --provider anthropic \ + --model claude-3-5-sonnet-latest \ + --prompt "Reply with the word ready." +``` + +The check command prints the HTTP status, elapsed milliseconds, and model text. + +## Benchmarks + +Run an AIME 2026 benchmark: + +```bash +cargo run -- bench aime2026 \ + --config config.yaml \ + --provider openai \ + --model gpt-4o-mini \ + --concurrency 4 \ + --limit 10 +``` + +Run a GPQA-Diamond benchmark: + +```bash +cargo run -- bench gpqa-diamond \ + --config config.yaml \ + --provider openai \ + --model gpt-4o-mini \ + --concurrency 4 \ + --limit 10 +``` + +Omit `--limit` to run all locally available cases. + +## RPM Testing + +Run an RPM test: + +```bash +cargo run -- rpm \ + --config config.yaml \ + --provider openai \ + --model gpt-4o-mini \ + --rpm 60 \ + --duration 60s \ + --prompt "Reply with pong." +``` + +Durations use `s` or `m`, such as `30s` or `5m`. + +## Reports + +Benchmark and RPM commands print a terminal summary with success counts, failures, latency percentiles, errors, and the report path. JSON reports are written under `reports/*.json`; the `reports` directory is ignored by git. + +Benchmark reports include `wrong_cases`, with each wrong case containing the case id, question, expected answer, extracted actual answer, and raw model output. RPM reports include request counts, target RPM, latency, and error counts. + +## Comparing Scores + +Use these results as relay benchmark signals, not absolute proof by themselves. To compare against official scores or another run, align the same dataset and source, prompt text, temperature, `max_tokens`, sample limit, and scoring logic. Differences in any of those inputs can make the reported accuracy diverge from official numbers or other benchmark harnesses.