| @@ -0,0 +1,431 @@ | |||
| # lq_token_test 使用文档 | |||
| `lq_token_test` 是一个 Rust 命令行工具,用来测试大模型中转站的协议兼容性、RPM 限流表现、并发表现,以及基于官方数据集的简单准确率表现。 | |||
| 当前支持: | |||
| - OpenAI-compatible 协议 | |||
| - Anthropic-compatible 协议 | |||
| - YAML 配置文件 | |||
| - AIME 2026 数据集 | |||
| - GPQA-Diamond 数据集 | |||
| - benchmark 准确率统计 | |||
| - RPM 限流模式测试 | |||
| - JSON 报告输出 | |||
| ## 1. 构建和基础检查 | |||
| ```bash | |||
| cargo fmt --check | |||
| cargo test | |||
| cargo clippy --all-targets -- -D warnings | |||
| cargo run -- --help | |||
| ``` | |||
| 编译 release 版本: | |||
| ```bash | |||
| cargo build --release | |||
| ``` | |||
| release 二进制在: | |||
| ```text | |||
| target/release/lq_token_test | |||
| ``` | |||
| ## 2. 配置文件 | |||
| 复制示例配置: | |||
| ```bash | |||
| cp config.example.yaml config.yaml | |||
| ``` | |||
| 示例: | |||
| ```yaml | |||
| default_provider: openai | |||
| providers: | |||
| openai: | |||
| protocol: openai | |||
| base_url: "https://relay.example.com/v1" | |||
| api_token: "${OPENAI_RELAY_TOKEN}" | |||
| default_model: "gpt-4o-mini" | |||
| anthropic: | |||
| protocol: anthropic | |||
| base_url: "https://relay.example.com" | |||
| api_token: "${ANTHROPIC_RELAY_TOKEN}" | |||
| default_model: "claude-3-5-sonnet-latest" | |||
| benchmarks: | |||
| data_dir: "data/benchmarks" | |||
| aime2026: | |||
| source: "huggingface:MathArena/aime_2026" | |||
| split: "train" | |||
| gpqa_diamond: | |||
| source: "huggingface:Idavidrein/gpqa" | |||
| split: "gpqa_diamond" | |||
| ``` | |||
| 设置 token: | |||
| ```bash | |||
| export OPENAI_RELAY_TOKEN="..." | |||
| export ANTHROPIC_RELAY_TOKEN="..." | |||
| ``` | |||
| 注意:工具只会解析当前使用的 provider token。比如只运行 `--provider anthropic` 时,不需要设置 `OPENAI_RELAY_TOKEN`。 | |||
| ## 3. 协议连通性测试 | |||
| OpenAI-compatible: | |||
| ```bash | |||
| cargo run -- check \ | |||
| --config config.yaml \ | |||
| --provider openai \ | |||
| --model gpt-4o-mini \ | |||
| --prompt "Reply with the word ready." | |||
| ``` | |||
| Anthropic-compatible: | |||
| ```bash | |||
| cargo run -- check \ | |||
| --config config.yaml \ | |||
| --provider anthropic \ | |||
| --model claude-3-5-sonnet-latest \ | |||
| --prompt "Reply with the word ready." | |||
| ``` | |||
| 如果不传 `--model`,会使用配置里的 `default_model`。 | |||
| ## 4. 下载数据集 | |||
| AIME 2026: | |||
| ```bash | |||
| cargo run -- dataset fetch aime2026 | |||
| ``` | |||
| 下载后会生成: | |||
| ```text | |||
| data/benchmarks/aime2026/aime2026.jsonl | |||
| data/benchmarks/aime2026/metadata.yaml | |||
| ``` | |||
| GPQA-Diamond: | |||
| ```bash | |||
| cargo run -- dataset fetch gpqa-diamond | |||
| ``` | |||
| 如果 Hugging Face 要求认证,先设置: | |||
| ```bash | |||
| export HF_TOKEN="..." | |||
| ``` | |||
| 数据集目录 `data/benchmarks/` 不会提交到 git。 | |||
| ## 5. 数据集 benchmark | |||
| AIME 2026 小样本: | |||
| ```bash | |||
| cargo run -- bench aime2026 \ | |||
| --config config.yaml \ | |||
| --provider anthropic \ | |||
| --limit 3 \ | |||
| --concurrency 1 | |||
| ``` | |||
| AIME 2026 完整测试: | |||
| ```bash | |||
| cargo run -- bench aime2026 \ | |||
| --config config.yaml \ | |||
| --provider anthropic \ | |||
| --concurrency 4 | |||
| ``` | |||
| GPQA-Diamond 小样本: | |||
| ```bash | |||
| cargo run -- bench gpqa-diamond \ | |||
| --config config.yaml \ | |||
| --provider anthropic \ | |||
| --limit 5 \ | |||
| --concurrency 1 | |||
| ``` | |||
| GPQA-Diamond 完整测试: | |||
| ```bash | |||
| cargo run -- bench gpqa-diamond \ | |||
| --config config.yaml \ | |||
| --provider anthropic \ | |||
| --concurrency 4 | |||
| ``` | |||
| benchmark 会输出: | |||
| - accuracy | |||
| - success / failed | |||
| - latency p50 / p95 / p99 | |||
| - errors | |||
| - wrong cases | |||
| - report 路径 | |||
| ## 6. RPM 测试模式 | |||
| 当前支持 6 种 RPM 模式: | |||
| ```text | |||
| sustained | |||
| burst | |||
| token-bucket | |||
| sliding-window | |||
| window-boundary | |||
| diagnose | |||
| ``` | |||
| ### 6.1 sustained:稳定 RPM 压测 | |||
| 适合测试服务是否能长期稳定承受目标 RPM。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode sustained \ | |||
| --provider anthropic \ | |||
| --rpm 120 \ | |||
| --duration 60s \ | |||
| --prompt "hello" | |||
| ``` | |||
| 逻辑: | |||
| ```text | |||
| 每 60 / rpm 秒启动一个请求 | |||
| 请求可以重叠 | |||
| 持续 duration | |||
| ``` | |||
| ### 6.2 burst:瞬时突发测试 | |||
| 适合测试瞬时并发、突发容量、初始桶容量。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode burst \ | |||
| --provider anthropic \ | |||
| --burst 120 \ | |||
| --prompt "hello" | |||
| ``` | |||
| 逻辑: | |||
| ```text | |||
| t=0 同时发送 burst 个请求 | |||
| ``` | |||
| ### 6.3 token-bucket:令牌桶测试 | |||
| 适合测试桶容量和 token refill 速率。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode token-bucket \ | |||
| --provider anthropic \ | |||
| --rpm 120 \ | |||
| --burst 120 \ | |||
| --probe-seconds 30 \ | |||
| --prompt "hello" | |||
| ``` | |||
| 逻辑: | |||
| ```text | |||
| 先 burst 打空桶 | |||
| 再按 rpm 推导的 refill 速率发送探针请求 | |||
| ``` | |||
| 例如: | |||
| ```text | |||
| rpm = 120 | |||
| refill rate = 2 req/s | |||
| ``` | |||
| ### 6.4 sliding-window:滑动窗口测试 | |||
| 适合测试滚动 60 秒窗口的恢复行为。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode sliding-window \ | |||
| --provider anthropic \ | |||
| --rpm 120 \ | |||
| --burst 120 \ | |||
| --probe-seconds 90 \ | |||
| --prompt "hello" | |||
| ``` | |||
| 逻辑: | |||
| ```text | |||
| t=0 burst | |||
| 之后每秒发送探针 | |||
| 观察是否接近 60 秒后恢复 | |||
| ``` | |||
| ### 6.5 window-boundary:固定窗口边界测试 | |||
| 适合测试 fixed window 是否在自然分钟边界重置额度。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode window-boundary \ | |||
| --provider anthropic \ | |||
| --rpm 120 \ | |||
| --burst 120 \ | |||
| --window-offset-ms 500 \ | |||
| --prompt "hello" | |||
| ``` | |||
| 逻辑: | |||
| ```text | |||
| 在分钟边界前发送一批请求 | |||
| 在分钟边界后发送一批请求 | |||
| 观察边界后是否明显恢复 | |||
| ``` | |||
| ### 6.6 diagnose:未知模式诊断 | |||
| 如果不知道后端是什么限流模式,可以用 diagnose。 | |||
| ```bash | |||
| cargo run -- rpm \ | |||
| --config config.yaml \ | |||
| --mode diagnose \ | |||
| --provider anthropic \ | |||
| --rpm 120 \ | |||
| --burst 120 \ | |||
| --probe-seconds 90 \ | |||
| --prompt "hello" | |||
| ``` | |||
| 报告中会给出 best-effort 推断: | |||
| ```text | |||
| token_bucket | |||
| fixed_window | |||
| sliding_window | |||
| unknown | |||
| ``` | |||
| 注意:真实大模型服务通常同时包含 RPM、TPM、并发数、队列长度、账号级限流、模型级限流、区域级限流等多种机制。诊断结果是强信号,不是绝对证明。 | |||
| ## 7. 报告文件 | |||
| benchmark 和 rpm 都会生成 JSON 报告: | |||
| ```text | |||
| reports/*.json | |||
| ``` | |||
| 查看最新报告: | |||
| ```bash | |||
| ls -lt reports | head | |||
| ``` | |||
| 格式化查看: | |||
| ```bash | |||
| jq . reports/<报告文件>.json | |||
| ``` | |||
| benchmark report 包含: | |||
| - benchmark | |||
| - provider | |||
| - model | |||
| - dataset | |||
| - run 参数 | |||
| - accuracy | |||
| - success / failed | |||
| - latency p50 / p95 / p99 | |||
| - errors | |||
| - wrong_cases | |||
| RPM report 包含: | |||
| - mode | |||
| - provider | |||
| - model | |||
| - target_rpm | |||
| - actual_rpm | |||
| - request counts | |||
| - latency p50 / p95 / p99 | |||
| - errors | |||
| - mode_detail | |||
| - diagnose inference | |||
| RPM report 不会保存原始 prompt,也不会保存 token。 | |||
| ## 8. 推荐测试顺序 | |||
| 第一次测试建议: | |||
| ```text | |||
| 1. check 单次请求 | |||
| 2. sustained 低 RPM | |||
| 3. burst 小突发 | |||
| 4. 数据集 limit 小样本 | |||
| 5. 提高 concurrency | |||
| 6. token-bucket / sliding-window / window-boundary | |||
| 7. diagnose | |||
| 8. 完整 benchmark | |||
| ``` | |||
| 示例: | |||
| ```bash | |||
| cargo run -- check --provider anthropic --prompt "hello" | |||
| cargo run -- rpm --mode sustained --provider anthropic --rpm 10 --duration 30s --prompt "hello" | |||
| cargo run -- rpm --mode burst --provider anthropic --burst 10 --prompt "hello" | |||
| cargo run -- bench aime2026 --provider anthropic --limit 3 --concurrency 1 | |||
| ``` | |||
| ## 9. 和官方结果对比 | |||
| 使用 benchmark 结果对比官方宣传分数时,需要尽量对齐: | |||
| - 数据集来源 | |||
| - 数据集版本 | |||
| - prompt 模板 | |||
| - temperature | |||
| - max_tokens | |||
| - 样本数量 | |||
| - 判分逻辑 | |||
| - 是否允许工具调用 | |||
| - 是否多次采样 | |||
| 当前工具的结果更适合作为: | |||
| ```text | |||
| 中转站 relay benchmark signal | |||
| ``` | |||
| 不要把单次结果直接当作官方 benchmark 的严格复现。 | |||