diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..323b55f --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,51 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Build Commands + +```bash +cargo build # debug build +cargo build --release # release build +cargo test # run all tests +cargo test # run a single test +cargo clippy --all-targets -- -D warnings # lint +cargo fmt --check # format check +cargo fmt # auto-format +``` + +Tracing is controlled via `RUST_LOG` env var (uses tracing-subscriber with env-filter). + +## Architecture + +This is a Rust CLI (`clap` derive) for testing LLM relay compatibility, benchmark accuracy, and RPM rate-limit behavior. It supports OpenAI-compatible and Anthropic-compatible relay protocols. + +### Module Layout + +- `cli.rs` — CLI definition and command dispatch. Contains all subcommand handlers (`check`, `dataset fetch`, `bench`, `rpm`). The RPM command builds a schedule of timed probes, executes them concurrently, then aggregates results into a mode-specific report. +- `config.rs` — YAML config loading with `${ENV_VAR}` expansion. Provider tokens are resolved lazily (only the requested provider's env vars need to be set). `AppConfig::load()` is the entry point. +- `runner.rs` — Thin orchestrator that dispatches to the correct protocol adapter and measures elapsed time. `run_model_request()` is the single call site for all LLM requests. +- `protocols/` — Protocol adapters (`openai.rs`, `anthropic.rs`). Each implements `send(client, request) -> ModelResponse`. Shared URL normalization and error extraction live in `protocols/mod.rs`. +- `benchmarks/` — Dataset fetching from Hugging Face (`fetch_dataset`), case loaders (`aime.rs`, `gpqa.rs`), and answer judging (`judge.rs`). AIME rows are normalized from HF API JSON to local JSONL. +- `metrics.rs` — HDR histogram-based latency tracking plus success/failure/accuracy counters. `Metrics::summary()` produces the final `MetricsSummary`. +- `rpm_modes.rs` — Schedule generators for each RPM test mode (sustained, burst, token-bucket, sliding-window, window-boundary, diagnose). Each returns a `Vec` with offsets and phase labels. +- `report.rs` — Report structs and JSON serialization. Reports are written to `reports/` directory. + +### Request Flow + +`Cli::parse()` → `cli::dispatch()` → builds `ModelRequest` → `runner::run_model_request()` → `protocols::{openai,anthropic}::send()` → returns `ModelResponse`. + +For RPM tests, requests are scheduled via `tokio::time::sleep_until` with `stream::buffer_unordered` for concurrency control. + +### Configuration + +Config is loaded from `config.yaml` (YAML). Values like `${ENV_NAME}` are expanded at load time. Provider tokens are expanded only when that provider is resolved, so unused providers don't require their env vars to be set. + +## Conventions + +- Rust edition 2024 +- Error handling: `anyhow::Result` for application errors, `thiserror` for typed errors in `config.rs` +- Async runtime: tokio multi-thread +- HTTP client: reqwest with rustls-tls (no OpenSSL dependency) +- API tokens are redacted in Debug impls +- Tests use `tempfile` for filesystem isolation and `wiremock` for HTTP mocking diff --git a/config.example.yaml b/config.example.yaml index f33bdcc..3424afe 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -19,10 +19,11 @@ providers: default_model: "claude-3-5-sonnet-latest" stream: true thinking: - enabled: false - type: "enabled" - budget_tokens: 10000 - display: "omitted" + enabled: true + type: adaptive # enabled | adaptive + budget_tokens: 10000 # Anthropic enabled 模式 , Opus 4.7 上是只能用 adaptive + effort。 + effort: high # Anthropic adaptive 模式可用 + display: summarized # summarized | omitted benchmarks: data_dir: "data/benchmarks"