| @@ -16,16 +16,17 @@ First phase: | |||
| - Manage relay base URLs, API tokens, default models, and provider settings. | |||
| - Support OpenAI-compatible chat requests. | |||
| - Add Anthropic-compatible request structure behind the same internal runner boundary. | |||
| - Provide CLI subcommands for single checks, benchmark smoke tests, and RPM/concurrency tests. | |||
| - Support local JSONL benchmark files. | |||
| - Include simple exact-match and GSM8K-style numeric answer judging. | |||
| - Provide CLI subcommands for single checks, official benchmark tests, and RPM/concurrency tests. | |||
| - Support the official GSM8K test split as the first benchmark dataset. | |||
| - Include GSM8K-style numeric answer judging. | |||
| - Report success count, error count, latency summaries, and benchmark accuracy. | |||
| Later phase: | |||
| - Add stricter benchmark profiles for comparing against official model reports. | |||
| - Record prompt template, dataset version, sampling parameters, model identity, and scoring method. | |||
| - Add larger public dataset ingestion only after the local JSONL format is stable. | |||
| - Add more official benchmarks such as MMLU-style multiple-choice tasks after GSM8K is stable. | |||
| - Add custom local JSONL ingestion as an extension, not as the first benchmark path. | |||
| ## Architecture | |||
| @@ -44,9 +45,9 @@ src/ | |||
| mod.rs | |||
| openai.rs | |||
| anthropic.rs | |||
| benchmark/ | |||
| benchmarks/ | |||
| mod.rs | |||
| dataset.rs | |||
| gsm8k.rs | |||
| judge.rs | |||
| ``` | |||
| @@ -56,7 +57,7 @@ Module responsibilities: | |||
| - `config`: Loads YAML config, resolves environment variable references, and validates provider settings. | |||
| - `protocols`: Converts internal request data into provider-specific HTTP requests and parses responses. | |||
| - `runner`: Executes one logical model request and returns response text, elapsed time, provider metadata, and errors. | |||
| - `benchmark`: Reads JSONL test cases, runs them through the runner, and judges answers. | |||
| - `benchmarks`: Loads official benchmark datasets, runs them through the runner, and judges answers. | |||
| - `metrics`: Aggregates latency, success rate, error distribution, and accuracy summaries. | |||
| - `main`: Wires the CLI, config, runner, and command handlers together. | |||
| @@ -66,13 +67,14 @@ Initial commands: | |||
| ```bash | |||
| lq_token_test check --config config.yaml --provider openai --model gpt-4o-mini --prompt "hello" | |||
| lq_token_test bench --config config.yaml --provider openai --dataset benchmarks/smoke.jsonl --concurrency 4 | |||
| lq_token_test bench gsm8k --config config.yaml --provider openai --model gpt-4o-mini --limit 100 --concurrency 4 | |||
| lq_token_test bench gsm8k --config config.yaml --provider anthropic --model claude-3-5-sonnet-latest --limit 100 --concurrency 4 | |||
| lq_token_test rpm --config config.yaml --provider openai --rpm 60 --duration 60s --prompt "hello" | |||
| ``` | |||
| The `check` command proves that a relay, token, model, and protocol shape work. | |||
| The `bench` command runs local JSONL cases and reports accuracy plus request metrics. | |||
| The `bench` command runs an official benchmark dataset and reports accuracy plus request metrics. | |||
| The `rpm` command sends repeated requests at a target rate and reports latency and error behavior. | |||
| @@ -97,30 +99,31 @@ providers: | |||
| default_model: "claude-3-5-sonnet-latest" | |||
| benchmarks: | |||
| smoke: | |||
| path: "benchmarks/smoke.jsonl" | |||
| kind: "qa_exact" | |||
| cache_dir: ".cache/lq_token_test/benchmarks" | |||
| gsm8k: | |||
| source: "official_openai_github" | |||
| split: "test" | |||
| ``` | |||
| API tokens should be allowed directly in YAML for local testing, but environment variable references are preferred. | |||
| ## Benchmark Format | |||
| ## Benchmark Data | |||
| Use JSONL so datasets can be streamed and edited without custom tooling. | |||
| First benchmark dataset: | |||
| Example: | |||
| - GSM8K official test split from OpenAI's `grade-school-math` dataset. | |||
| - Source repository: `openai/grade-school-math`. | |||
| - Source file: `grade_school_math/data/test.jsonl`. | |||
| - Expected format: each line contains `question` and `answer`. | |||
| - The final answer is extracted from the official `answer` field using the GSM8K `#### <answer>` convention. | |||
| ```jsonl | |||
| {"id":"smoke_001","question":"What is 2 + 2?","answer":"4","kind":"exact"} | |||
| {"id":"gsm8k_001","question":"Natalia sold clips and earned 72 dollars. How much did she earn?","answer":"72","kind":"gsm8k"} | |||
| ``` | |||
| The CLI should be able to download and cache the official test file when network access is available. It should also accept a local official GSM8K `test.jsonl` path for offline or pinned-version runs. | |||
| Judging rules: | |||
| - `exact`: normalize whitespace and compare the full answer. | |||
| - `gsm8k`: extract the final numeric answer from model output and compare with the expected answer. | |||
| The first phase accuracy score is an internal evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset version, prompt template, sample count, temperature, and scoring method are aligned with the official report. | |||
| The first phase accuracy score is an official-dataset relay evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset commit, prompt template, sample count, temperature, and scoring method are aligned with the official report. | |||
| ## Dependencies | |||
| @@ -149,7 +152,7 @@ The CLI should surface concise user-facing errors: | |||
| - Unsupported protocol. | |||
| - HTTP status failures. | |||
| - Provider response parse failures. | |||
| - Invalid benchmark JSONL lines. | |||
| - Invalid official benchmark data lines. | |||
| Benchmark and RPM commands should continue after per-request failures and include failures in the final summary. | |||
| @@ -158,8 +161,7 @@ Benchmark and RPM commands should continue after per-request failures and includ | |||
| Initial tests should cover: | |||
| - YAML config loading and environment variable expansion. | |||
| - JSONL benchmark parsing. | |||
| - Exact-match judging. | |||
| - Official GSM8K test data parsing. | |||
| - GSM8K-style numeric extraction. | |||
| - Metrics aggregation. | |||
| @@ -168,5 +170,7 @@ Network tests should be kept opt-in because they require real relay credentials. | |||
| ## First Implementation Decisions | |||
| - OpenAI-compatible support starts with `/chat/completions`. | |||
| - Anthropic support includes a real adapter boundary and request shape in the first pass, but end-to-end verification can wait until a real Anthropic-compatible relay config is available. | |||
| - Benchmark cases use a minimal default prompt template: ask the question and request only the final answer. Dataset-specific templates can be added later. | |||
| - Anthropic support includes a real adapter boundary and request shape in the first pass. | |||
| - The first benchmark target is the official GSM8K test split. | |||
| - Benchmark prompts ask the model to solve the problem and end with a single final numeric answer. The exact prompt template is recorded in benchmark output. | |||
| - Benchmark runs should record provider, model, dataset source, dataset split, optional dataset commit, limit, concurrency, temperature, max tokens, accuracy, latency percentiles, and error summary. | |||