ソースを参照

docs: define benchmark dataset download flow

main
orangels 1ヶ月前
コミット
61446dcf9f
1個のファイルの変更46行の追加25行の削除
  1. +46
    -25
      docs/superpowers/specs/2026-05-06-lq-token-test-design.md

+ 46
- 25
docs/superpowers/specs/2026-05-06-lq-token-test-design.md ファイルの表示

@@ -17,15 +17,17 @@ First phase:
- Support OpenAI-compatible chat requests.
- Add Anthropic-compatible request structure behind the same internal runner boundary.
- Provide CLI subcommands for single checks, official benchmark tests, and RPM/concurrency tests.
- Support the official GSM8K test split as the first benchmark dataset.
- Include GSM8K-style numeric answer judging.
- Support AIME 2026 and GPQA-Diamond as the first benchmark datasets.
- Provide dataset download commands so users can fetch official datasets into local storage before running benchmarks.
- Keep downloaded datasets out of git.
- Include numeric answer judging for AIME and multiple-choice judging for GPQA-Diamond.
- Report success count, error count, latency summaries, and benchmark accuracy.

Later phase:

- Add stricter benchmark profiles for comparing against official model reports.
- Record prompt template, dataset version, sampling parameters, model identity, and scoring method.
- Add more official benchmarks such as MMLU-style multiple-choice tasks after GSM8K is stable.
- Add more official benchmarks such as HLE text-only or HMMT after AIME 2026 and GPQA-Diamond are stable.
- Add custom local JSONL ingestion as an extension, not as the first benchmark path.

## Architecture
@@ -47,7 +49,8 @@ src/
anthropic.rs
benchmarks/
mod.rs
gsm8k.rs
aime.rs
gpqa.rs
judge.rs
```

@@ -67,13 +70,17 @@ Initial commands:

```bash
lq_token_test check --config config.yaml --provider openai --model gpt-4o-mini --prompt "hello"
lq_token_test bench gsm8k --config config.yaml --provider openai --model gpt-4o-mini --limit 100 --concurrency 4
lq_token_test bench gsm8k --config config.yaml --provider anthropic --model claude-3-5-sonnet-latest --limit 100 --concurrency 4
lq_token_test dataset fetch aime2026
lq_token_test dataset fetch gpqa-diamond
lq_token_test bench aime2026 --config config.yaml --provider openai --model gpt-4o-mini --concurrency 4
lq_token_test bench gpqa-diamond --config config.yaml --provider anthropic --model claude-3-5-sonnet-latest --limit 100 --concurrency 4
lq_token_test rpm --config config.yaml --provider openai --rpm 60 --duration 60s --prompt "hello"
```

The `check` command proves that a relay, token, model, and protocol shape work.

The `dataset fetch` command downloads an official dataset into local storage and writes source metadata.

The `bench` command runs an official benchmark dataset and reports accuracy plus request metrics.

The `rpm` command sends repeated requests at a target rate and reports latency and error behavior.
@@ -99,31 +106,40 @@ providers:
default_model: "claude-3-5-sonnet-latest"

benchmarks:
cache_dir: ".cache/lq_token_test/benchmarks"
gsm8k:
source: "official_openai_github"
split: "test"
data_dir: "data/benchmarks"
aime2026:
source: "huggingface:MathArena/aime_2026"
split: "train"
gpqa_diamond:
source: "huggingface:Idavidrein/gpqa"
split: "gpqa_diamond"
```

API tokens should be allowed directly in YAML for local testing, but environment variable references are preferred.

## Benchmark Data

First benchmark dataset:
First benchmark datasets:

- GSM8K official test split from OpenAI's `grade-school-math` dataset.
- Source repository: `openai/grade-school-math`.
- Source file: `grade_school_math/data/test.jsonl`.
- Expected format: each line contains `question` and `answer`.
- The final answer is extracted from the official `answer` field using the GSM8K `#### <answer>` convention.
- AIME 2026 from Hugging Face dataset `MathArena/aime_2026`.
- Format: parquet.
- Size: 30 rows.
- Fields: `problem_idx`, `problem`, `answer`.
- License: CC BY-NC-SA 4.0.
- Purpose: current math reasoning benchmark with simple final numeric judging.
- GPQA-Diamond from Hugging Face dataset `Idavidrein/gpqa`.
- Format: CSV.
- Access: requires accepting the dataset conditions on Hugging Face; downloads should support `HF_TOKEN`.
- Purpose: current science reasoning benchmark with multiple-choice judging.

The CLI should be able to download and cache the official test file when network access is available. It should also accept a local official GSM8K `test.jsonl` path for offline or pinned-version runs.
Downloaded data should live under `data/benchmarks/` by default and should not be committed to git. Each download writes a `metadata.yaml` with dataset name, source, split, download time, file hash, row count, and license note. Benchmark commands should read from local files and tell the user to run `dataset fetch` if required data is missing.

Judging rules:

- `gsm8k`: extract the final numeric answer from model output and compare with the expected answer.
- `aime2026`: extract the final integer answer from model output and compare with the official `answer`.
- `gpqa-diamond`: ask the model to answer with one option letter and compare against the official correct option.

The first phase accuracy score is an official-dataset relay evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset commit, prompt template, sample count, temperature, and scoring method are aligned with the official report.
The first phase accuracy score is an official-dataset relay evaluation signal. It should not be presented as matching or disproving official model accuracy until dataset revision, prompt template, sample count, temperature, and scoring method are aligned with the official report.

## Dependencies

@@ -138,7 +154,7 @@ Recommended crates:
- `tracing`, `tracing-subscriber`: logs.
- `indicatif`: progress display for benchmark and RPM runs.
- `hdrhistogram`: latency percentiles.
- `regex`: answer extraction for GSM8K-style judging.
- `regex`: answer extraction for numeric and multiple-choice judging.

Prefer `reqwest` with Rustls TLS. Avoid provider SDKs in the first phase so protocol compatibility remains transparent and easy to inspect.

@@ -153,6 +169,8 @@ The CLI should surface concise user-facing errors:
- HTTP status failures.
- Provider response parse failures.
- Invalid official benchmark data lines.
- Missing local benchmark data.
- Dataset download failures.

Benchmark and RPM commands should continue after per-request failures and include failures in the final summary.

@@ -161,8 +179,10 @@ Benchmark and RPM commands should continue after per-request failures and includ
Initial tests should cover:

- YAML config loading and environment variable expansion.
- Official GSM8K test data parsing.
- GSM8K-style numeric extraction.
- AIME 2026 data parsing.
- GPQA-Diamond data parsing.
- AIME-style numeric extraction.
- GPQA multiple-choice extraction.
- Metrics aggregation.

Network tests should be kept opt-in because they require real relay credentials.
@@ -171,6 +191,7 @@ Network tests should be kept opt-in because they require real relay credentials.

- OpenAI-compatible support starts with `/chat/completions`.
- Anthropic support includes a real adapter boundary and request shape in the first pass.
- The first benchmark target is the official GSM8K test split.
- Benchmark prompts ask the model to solve the problem and end with a single final numeric answer. The exact prompt template is recorded in benchmark output.
- Benchmark runs should record provider, model, dataset source, dataset split, optional dataset commit, limit, concurrency, temperature, max tokens, accuracy, latency percentiles, and error summary.
- The first benchmark targets are AIME 2026 and GPQA-Diamond.
- Datasets are downloaded by command into local storage and are not committed to git.
- Benchmark prompts ask the model to use the answer format required by the dataset. The exact prompt template is recorded in benchmark output.
- Benchmark runs should record provider, model, dataset source, dataset split, optional dataset revision, limit, concurrency, temperature, max tokens, accuracy, latency percentiles, and error summary.

読み込み中…
キャンセル
保存