From f6b09d00a41dbfb6e8168f93baa8c9cbdd4a9cf7 Mon Sep 17 00:00:00 2001 From: orangels Date: Wed, 6 May 2026 21:16:07 +0800 Subject: [PATCH] docs: plan rpm limiter modes --- ...-05-06-rpm-limiter-modes-implementation.md | 398 ++++++++++++++++++ 1 file changed, 398 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-06-rpm-limiter-modes-implementation.md diff --git a/docs/superpowers/plans/2026-05-06-rpm-limiter-modes-implementation.md b/docs/superpowers/plans/2026-05-06-rpm-limiter-modes-implementation.md new file mode 100644 index 0000000..943cd9a --- /dev/null +++ b/docs/superpowers/plans/2026-05-06-rpm-limiter-modes-implementation.md @@ -0,0 +1,398 @@ +# RPM Limiter Modes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extend `rpm` testing from one sustained schedule into six explicit limiter test modes: `sustained`, `burst`, `token-bucket`, `sliding-window`, `window-boundary`, and `diagnose`. + +**Architecture:** Keep mode-specific scheduling and inference in a focused module, while `cli` remains responsible for argument parsing and command dispatch. Reports should preserve the current RPM summary shape and add mode-specific detail without storing raw prompts or secrets. + +**Tech Stack:** Rust 2024, existing `clap`, `tokio`, `futures`, `serde`, `chrono`, `hdrhistogram`, `anyhow`. + +--- + +## File Structure + +- Modify `src/cli.rs`: add `--mode`, mode-specific flags, dispatch to mode runner, and keep current sustained behavior as default. +- Create `src/rpm_modes.rs`: mode enum, schedules, probe plans, limiter inference, and unit tests. +- Modify `src/report.rs`: extend RPM reports with mode, optional burst/probe/window/diagnose details, actual RPM, and inferred limiter. +- Modify `src/main.rs`: register `rpm_modes` module. +- Modify `README.md`: document RPM modes and example commands. + +## Mode Semantics + +- `sustained`: current default. Starts one request every `60 / rpm` seconds for `duration`. Requests may overlap. +- `burst`: starts `burst` requests at `t=0` and reports immediate success/failure/latency/error behavior. +- `token-bucket`: starts `burst` requests at `t=0`, then probes refill behavior for `probe_seconds`. Probe rate defaults to the expected refill rate from `rpm`, rounded up to at least one probe per second. +- `sliding-window`: starts `burst` requests at `t=0`, then sends low-rate probes for `probe_seconds` to observe whether recovery happens near the rolling 60 second boundary. +- `window-boundary`: waits until the next minute boundary test point, sends one batch before the boundary and one batch after it. This tests fixed-window reset behavior. +- `diagnose`: runs a bounded combined probe using burst, refill probes, and boundary signals, then writes a best-effort inference: `token_bucket`, `fixed_window`, `sliding_window`, or `unknown`. + +## CLI Shape + +Keep the old command valid: + +```bash +cargo run -- rpm --provider anthropic --rpm 120 --duration 60s --prompt "hello" +``` + +Equivalent explicit command: + +```bash +cargo run -- rpm --mode sustained --provider anthropic --rpm 120 --duration 60s --prompt "hello" +``` + +New commands: + +```bash +cargo run -- rpm --mode burst --provider anthropic --burst 120 --prompt "hello" +cargo run -- rpm --mode token-bucket --provider anthropic --rpm 120 --burst 120 --probe-seconds 30 --prompt "hello" +cargo run -- rpm --mode sliding-window --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" +cargo run -- rpm --mode window-boundary --provider anthropic --rpm 120 --burst 120 --prompt "hello" +cargo run -- rpm --mode diagnose --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" +``` + +Argument defaults: + +- `--mode sustained` +- `--duration 60s`, required only by `sustained` +- `--burst ` default for burst-style modes when omitted +- `--probe-seconds 90` for `sliding-window` and `diagnose` +- `--probe-seconds 30` for `token-bucket` +- `--window-offset-ms 500` for `window-boundary`, meaning send before/after batches around the next minute boundary +- `--concurrency` should cap in-flight requests for all modes; default can be `burst` or a safe high value, but must never be zero + +## Report Shape + +Extend `RpmReport` with: + +```json +{ + "mode": "token-bucket", + "provider": "anthropic", + "model": "claude-test", + "run": { + "started_at": "2026-05-06T15:30:12Z", + "duration_ms": 60000, + "target_rpm": 120, + "actual_rpm": 118.5, + "temperature": 0.0, + "max_tokens": 1024 + }, + "summary": { + "actual_requests": 120, + "success": 118, + "failure": 2, + "latency_ms": { + "p50": 800, + "p95": 1500, + "p99": 2200 + } + }, + "mode_detail": { + "burst": { + "sent": 120, + "success": 118, + "failure": 2 + }, + "refill_probe": [ + { + "second": 1, + "sent": 2, + "success": 2, + "failure": 0 + } + ], + "inference": { + "likely_limiter": "token_bucket", + "confidence": "medium", + "signals": [ + "burst accepted most initial requests", + "probe success approximated 2.00 req/s refill" + ] + } + }, + "errors": [] +} +``` + +Rules: + +- Do not serialize raw prompt. +- Do not serialize tokens. +- Include `mode` at top level. +- Include `actual_rpm` for every mode where duration is meaningful. +- Include mode detail only when relevant. +- Existing sustained reports remain easy to read. + +## Task 1: Mode Types And Scheduling + +**Files:** +- Create: `src/rpm_modes.rs` +- Modify: `src/main.rs` + +- [ ] **Step 1: Add mode enum and parser** + +Implement `RpmMode` with `clap::ValueEnum`: + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum)] +#[value(rename_all = "kebab-case")] +pub enum RpmMode { + Sustained, + Burst, + TokenBucket, + SlidingWindow, + WindowBoundary, + Diagnose, +} +``` + +- [ ] **Step 2: Add scheduling helpers** + +Implement: + +```rust +pub fn sustained_schedule(duration: Duration, rpm: u32) -> Vec; +pub fn burst_schedule(burst: u32) -> Vec; +pub fn token_bucket_schedule(rpm: u32, burst: u32, probe_seconds: u64) -> Vec; +pub fn sliding_window_schedule(burst: u32, probe_seconds: u64) -> Vec; +pub fn window_boundary_plan(now: DateTime, burst: u32, offset_ms: u64) -> WindowBoundaryPlan; +``` + +Use explicit structs for `ScheduledProbe` and `WindowBoundaryPlan`; include phase labels such as `burst`, `refill_probe`, `sliding_probe`, `before_boundary`, and `after_boundary`. + +- [ ] **Step 3: Add schedule tests** + +Test: + +- 120 RPM sustained creates starts at 0ms, 500ms, 1000ms for a short duration. +- Burst 5 creates five 0ms starts. +- Token bucket 120 RPM with 120 burst and 2 probe seconds has 120 burst starts plus about 4 refill probes. +- Sliding window probe lasts through the requested probe seconds. +- Window boundary plan places before and after batches around a minute boundary. + +- [ ] **Step 4: Register module** + +Add `mod rpm_modes;` to `src/main.rs`. + +- [ ] **Step 5: Verify and commit** + +Run: + +```bash +cargo fmt +cargo test rpm_modes +``` + +Commit: + +```bash +git add src/rpm_modes.rs src/main.rs +git commit -m "feat: add rpm mode schedules" +``` + +## Task 2: CLI Wiring And Mode Execution + +**Files:** +- Modify: `src/cli.rs` +- Modify: `src/rpm_modes.rs` + +- [ ] **Step 1: Extend CLI args** + +Add to `Command::Rpm`: + +```rust +#[arg(long, value_enum, default_value_t = RpmMode::Sustained)] +mode: RpmMode, +#[arg(long)] +burst: Option, +#[arg(long)] +probe_seconds: Option, +#[arg(long, default_value_t = 500)] +window_offset_ms: u64, +#[arg(long)] +concurrency: Option, +``` + +Keep existing `rpm`, `duration`, and `prompt` arguments compatible. `duration` may remain a string with default `60s`; non-sustained modes can ignore it unless needed. + +- [ ] **Step 2: Build a request runner helper** + +Refactor RPM execution so all modes share: + +```rust +async fn run_scheduled_requests( + protocol: ProtocolKind, + request: ModelRequest, + starts: Vec, + max_in_flight: usize, +) -> Vec; +``` + +Each result should retain phase/second metadata for mode-detail reporting. + +- [ ] **Step 3: Implement mode execution** + +Mode behavior: + +- `sustained`: use current behavior with `sustained_schedule`. +- `burst`: use `burst_schedule`. +- `token-bucket`: burst at 0, then refill probes based on `rpm`. +- `sliding-window`: burst at 0, then one probe per second until `probe_seconds`. +- `window-boundary`: sleep until planned before-boundary start, send before batch, then after batch. +- `diagnose`: run bounded burst/refill/boundary probes and produce inference. + +- [ ] **Step 4: Add CLI behavior tests** + +Add tests for: + +- Old RPM command defaults to `sustained`. +- `--mode token-bucket` parses. +- `--mode window-boundary --window-offset-ms 250` parses. +- Invalid zero `--burst` and zero `--rpm` are rejected by execution validation. + +- [ ] **Step 5: Verify and commit** + +Run: + +```bash +cargo fmt +cargo test cli::tests rpm_modes +cargo test +``` + +Commit: + +```bash +git add src/cli.rs src/rpm_modes.rs +git commit -m "feat: run rpm limiter modes" +``` + +## Task 3: Report Extensions And Inference + +**Files:** +- Modify: `src/report.rs` +- Modify: `src/cli.rs` +- Modify: `src/rpm_modes.rs` + +- [ ] **Step 1: Extend report structs** + +Add: + +```rust +pub enum LimiterInferenceKind { + TokenBucket, + FixedWindow, + SlidingWindow, + Unknown, +} + +pub struct LimiterInferenceReport { + pub likely_limiter: LimiterInferenceKind, + pub confidence: String, + pub signals: Vec, +} + +pub struct RpmModeDetailReport { + pub burst: Option, + pub refill_probe: Vec, + pub sliding_probe: Vec, + pub window_boundary: Option, + pub inference: Option, +} +``` + +Use serde rename attributes so JSON uses snake_case values like `token_bucket`. + +- [ ] **Step 2: Summarize phases** + +Add helpers to group scheduled results by phase and second: + +- burst sent/success/failure +- refill probe per second +- sliding probe per second +- boundary before/after sent/success/failure + +- [ ] **Step 3: Implement diagnose inference** + +Best-effort rules: + +- If boundary after batch succeeds much more than before batch, infer `fixed_window` with medium confidence. +- If refill probes recover near `rpm / 60` requests per second after burst, infer `token_bucket` with medium confidence. +- If probes mostly fail until near 60 seconds after burst, infer `sliding_window` with medium confidence. +- Otherwise infer `unknown` with low confidence. + +Do not overclaim; include signals in report. + +- [ ] **Step 4: Add report tests** + +Test: + +- RPM report serializes `mode`. +- RPM report does not serialize raw prompt. +- Token bucket report includes burst and refill probe fields. +- Diagnose inference serializes expected snake_case limiter value. + +- [ ] **Step 5: Verify and commit** + +Run: + +```bash +cargo fmt +cargo test report rpm_modes +cargo test +``` + +Commit: + +```bash +git add src/report.rs src/cli.rs src/rpm_modes.rs +git commit -m "feat: report rpm limiter mode details" +``` + +## Task 4: Documentation And Final Verification + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Document six modes** + +Add a section for: + +- `sustained`: stable RPM. +- `burst`: instant burst capacity. +- `token-bucket`: burst plus refill probe. +- `sliding-window`: rolling 60 second recovery probe. +- `window-boundary`: fixed minute boundary probe. +- `diagnose`: combined unknown-mode diagnosis. + +- [ ] **Step 2: Document interpretation caveats** + +Mention that real LLM backends often combine RPM, TPM, concurrency, account-level, model-level, and region-level limits. Reports are strong signals, not perfect proof. + +- [ ] **Step 3: Final verification** + +Run: + +```bash +cargo fmt --check +cargo test +cargo clippy --all-targets -- -D warnings +cargo run -- rpm --help +``` + +Do not run live network tests unless the user explicitly asks. + +- [ ] **Step 4: Commit** + +```bash +git add README.md +git commit -m "docs: document rpm limiter modes" +``` + +## Self-Review + +- Spec coverage: covers all six requested modes and keeps existing sustained behavior compatible. +- Document scan: no unresolved planning gaps. +- Security: reports must not store prompts or tokens. +- Testability: schedule, parser, report, and inference tests avoid network calls.