| @@ -0,0 +1,398 @@ | |||
| # RPM Limiter Modes Implementation Plan | |||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | |||
| **Goal:** Extend `rpm` testing from one sustained schedule into six explicit limiter test modes: `sustained`, `burst`, `token-bucket`, `sliding-window`, `window-boundary`, and `diagnose`. | |||
| **Architecture:** Keep mode-specific scheduling and inference in a focused module, while `cli` remains responsible for argument parsing and command dispatch. Reports should preserve the current RPM summary shape and add mode-specific detail without storing raw prompts or secrets. | |||
| **Tech Stack:** Rust 2024, existing `clap`, `tokio`, `futures`, `serde`, `chrono`, `hdrhistogram`, `anyhow`. | |||
| --- | |||
| ## File Structure | |||
| - Modify `src/cli.rs`: add `--mode`, mode-specific flags, dispatch to mode runner, and keep current sustained behavior as default. | |||
| - Create `src/rpm_modes.rs`: mode enum, schedules, probe plans, limiter inference, and unit tests. | |||
| - Modify `src/report.rs`: extend RPM reports with mode, optional burst/probe/window/diagnose details, actual RPM, and inferred limiter. | |||
| - Modify `src/main.rs`: register `rpm_modes` module. | |||
| - Modify `README.md`: document RPM modes and example commands. | |||
| ## Mode Semantics | |||
| - `sustained`: current default. Starts one request every `60 / rpm` seconds for `duration`. Requests may overlap. | |||
| - `burst`: starts `burst` requests at `t=0` and reports immediate success/failure/latency/error behavior. | |||
| - `token-bucket`: starts `burst` requests at `t=0`, then probes refill behavior for `probe_seconds`. Probe rate defaults to the expected refill rate from `rpm`, rounded up to at least one probe per second. | |||
| - `sliding-window`: starts `burst` requests at `t=0`, then sends low-rate probes for `probe_seconds` to observe whether recovery happens near the rolling 60 second boundary. | |||
| - `window-boundary`: waits until the next minute boundary test point, sends one batch before the boundary and one batch after it. This tests fixed-window reset behavior. | |||
| - `diagnose`: runs a bounded combined probe using burst, refill probes, and boundary signals, then writes a best-effort inference: `token_bucket`, `fixed_window`, `sliding_window`, or `unknown`. | |||
| ## CLI Shape | |||
| Keep the old command valid: | |||
| ```bash | |||
| cargo run -- rpm --provider anthropic --rpm 120 --duration 60s --prompt "hello" | |||
| ``` | |||
| Equivalent explicit command: | |||
| ```bash | |||
| cargo run -- rpm --mode sustained --provider anthropic --rpm 120 --duration 60s --prompt "hello" | |||
| ``` | |||
| New commands: | |||
| ```bash | |||
| cargo run -- rpm --mode burst --provider anthropic --burst 120 --prompt "hello" | |||
| cargo run -- rpm --mode token-bucket --provider anthropic --rpm 120 --burst 120 --probe-seconds 30 --prompt "hello" | |||
| cargo run -- rpm --mode sliding-window --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" | |||
| cargo run -- rpm --mode window-boundary --provider anthropic --rpm 120 --burst 120 --prompt "hello" | |||
| cargo run -- rpm --mode diagnose --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" | |||
| ``` | |||
| Argument defaults: | |||
| - `--mode sustained` | |||
| - `--duration 60s`, required only by `sustained` | |||
| - `--burst <rpm>` default for burst-style modes when omitted | |||
| - `--probe-seconds 90` for `sliding-window` and `diagnose` | |||
| - `--probe-seconds 30` for `token-bucket` | |||
| - `--window-offset-ms 500` for `window-boundary`, meaning send before/after batches around the next minute boundary | |||
| - `--concurrency` should cap in-flight requests for all modes; default can be `burst` or a safe high value, but must never be zero | |||
| ## Report Shape | |||
| Extend `RpmReport` with: | |||
| ```json | |||
| { | |||
| "mode": "token-bucket", | |||
| "provider": "anthropic", | |||
| "model": "claude-test", | |||
| "run": { | |||
| "started_at": "2026-05-06T15:30:12Z", | |||
| "duration_ms": 60000, | |||
| "target_rpm": 120, | |||
| "actual_rpm": 118.5, | |||
| "temperature": 0.0, | |||
| "max_tokens": 1024 | |||
| }, | |||
| "summary": { | |||
| "actual_requests": 120, | |||
| "success": 118, | |||
| "failure": 2, | |||
| "latency_ms": { | |||
| "p50": 800, | |||
| "p95": 1500, | |||
| "p99": 2200 | |||
| } | |||
| }, | |||
| "mode_detail": { | |||
| "burst": { | |||
| "sent": 120, | |||
| "success": 118, | |||
| "failure": 2 | |||
| }, | |||
| "refill_probe": [ | |||
| { | |||
| "second": 1, | |||
| "sent": 2, | |||
| "success": 2, | |||
| "failure": 0 | |||
| } | |||
| ], | |||
| "inference": { | |||
| "likely_limiter": "token_bucket", | |||
| "confidence": "medium", | |||
| "signals": [ | |||
| "burst accepted most initial requests", | |||
| "probe success approximated 2.00 req/s refill" | |||
| ] | |||
| } | |||
| }, | |||
| "errors": [] | |||
| } | |||
| ``` | |||
| Rules: | |||
| - Do not serialize raw prompt. | |||
| - Do not serialize tokens. | |||
| - Include `mode` at top level. | |||
| - Include `actual_rpm` for every mode where duration is meaningful. | |||
| - Include mode detail only when relevant. | |||
| - Existing sustained reports remain easy to read. | |||
| ## Task 1: Mode Types And Scheduling | |||
| **Files:** | |||
| - Create: `src/rpm_modes.rs` | |||
| - Modify: `src/main.rs` | |||
| - [ ] **Step 1: Add mode enum and parser** | |||
| Implement `RpmMode` with `clap::ValueEnum`: | |||
| ```rust | |||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum)] | |||
| #[value(rename_all = "kebab-case")] | |||
| pub enum RpmMode { | |||
| Sustained, | |||
| Burst, | |||
| TokenBucket, | |||
| SlidingWindow, | |||
| WindowBoundary, | |||
| Diagnose, | |||
| } | |||
| ``` | |||
| - [ ] **Step 2: Add scheduling helpers** | |||
| Implement: | |||
| ```rust | |||
| pub fn sustained_schedule(duration: Duration, rpm: u32) -> Vec<Duration>; | |||
| pub fn burst_schedule(burst: u32) -> Vec<Duration>; | |||
| pub fn token_bucket_schedule(rpm: u32, burst: u32, probe_seconds: u64) -> Vec<ScheduledProbe>; | |||
| pub fn sliding_window_schedule(burst: u32, probe_seconds: u64) -> Vec<ScheduledProbe>; | |||
| pub fn window_boundary_plan(now: DateTime<Utc>, burst: u32, offset_ms: u64) -> WindowBoundaryPlan; | |||
| ``` | |||
| Use explicit structs for `ScheduledProbe` and `WindowBoundaryPlan`; include phase labels such as `burst`, `refill_probe`, `sliding_probe`, `before_boundary`, and `after_boundary`. | |||
| - [ ] **Step 3: Add schedule tests** | |||
| Test: | |||
| - 120 RPM sustained creates starts at 0ms, 500ms, 1000ms for a short duration. | |||
| - Burst 5 creates five 0ms starts. | |||
| - Token bucket 120 RPM with 120 burst and 2 probe seconds has 120 burst starts plus about 4 refill probes. | |||
| - Sliding window probe lasts through the requested probe seconds. | |||
| - Window boundary plan places before and after batches around a minute boundary. | |||
| - [ ] **Step 4: Register module** | |||
| Add `mod rpm_modes;` to `src/main.rs`. | |||
| - [ ] **Step 5: Verify and commit** | |||
| Run: | |||
| ```bash | |||
| cargo fmt | |||
| cargo test rpm_modes | |||
| ``` | |||
| Commit: | |||
| ```bash | |||
| git add src/rpm_modes.rs src/main.rs | |||
| git commit -m "feat: add rpm mode schedules" | |||
| ``` | |||
| ## Task 2: CLI Wiring And Mode Execution | |||
| **Files:** | |||
| - Modify: `src/cli.rs` | |||
| - Modify: `src/rpm_modes.rs` | |||
| - [ ] **Step 1: Extend CLI args** | |||
| Add to `Command::Rpm`: | |||
| ```rust | |||
| #[arg(long, value_enum, default_value_t = RpmMode::Sustained)] | |||
| mode: RpmMode, | |||
| #[arg(long)] | |||
| burst: Option<u32>, | |||
| #[arg(long)] | |||
| probe_seconds: Option<u64>, | |||
| #[arg(long, default_value_t = 500)] | |||
| window_offset_ms: u64, | |||
| #[arg(long)] | |||
| concurrency: Option<usize>, | |||
| ``` | |||
| Keep existing `rpm`, `duration`, and `prompt` arguments compatible. `duration` may remain a string with default `60s`; non-sustained modes can ignore it unless needed. | |||
| - [ ] **Step 2: Build a request runner helper** | |||
| Refactor RPM execution so all modes share: | |||
| ```rust | |||
| async fn run_scheduled_requests( | |||
| protocol: ProtocolKind, | |||
| request: ModelRequest, | |||
| starts: Vec<ScheduledRequest>, | |||
| max_in_flight: usize, | |||
| ) -> Vec<ScheduledResult>; | |||
| ``` | |||
| Each result should retain phase/second metadata for mode-detail reporting. | |||
| - [ ] **Step 3: Implement mode execution** | |||
| Mode behavior: | |||
| - `sustained`: use current behavior with `sustained_schedule`. | |||
| - `burst`: use `burst_schedule`. | |||
| - `token-bucket`: burst at 0, then refill probes based on `rpm`. | |||
| - `sliding-window`: burst at 0, then one probe per second until `probe_seconds`. | |||
| - `window-boundary`: sleep until planned before-boundary start, send before batch, then after batch. | |||
| - `diagnose`: run bounded burst/refill/boundary probes and produce inference. | |||
| - [ ] **Step 4: Add CLI behavior tests** | |||
| Add tests for: | |||
| - Old RPM command defaults to `sustained`. | |||
| - `--mode token-bucket` parses. | |||
| - `--mode window-boundary --window-offset-ms 250` parses. | |||
| - Invalid zero `--burst` and zero `--rpm` are rejected by execution validation. | |||
| - [ ] **Step 5: Verify and commit** | |||
| Run: | |||
| ```bash | |||
| cargo fmt | |||
| cargo test cli::tests rpm_modes | |||
| cargo test | |||
| ``` | |||
| Commit: | |||
| ```bash | |||
| git add src/cli.rs src/rpm_modes.rs | |||
| git commit -m "feat: run rpm limiter modes" | |||
| ``` | |||
| ## Task 3: Report Extensions And Inference | |||
| **Files:** | |||
| - Modify: `src/report.rs` | |||
| - Modify: `src/cli.rs` | |||
| - Modify: `src/rpm_modes.rs` | |||
| - [ ] **Step 1: Extend report structs** | |||
| Add: | |||
| ```rust | |||
| pub enum LimiterInferenceKind { | |||
| TokenBucket, | |||
| FixedWindow, | |||
| SlidingWindow, | |||
| Unknown, | |||
| } | |||
| pub struct LimiterInferenceReport { | |||
| pub likely_limiter: LimiterInferenceKind, | |||
| pub confidence: String, | |||
| pub signals: Vec<String>, | |||
| } | |||
| pub struct RpmModeDetailReport { | |||
| pub burst: Option<PhaseSummaryReport>, | |||
| pub refill_probe: Vec<ProbeSecondReport>, | |||
| pub sliding_probe: Vec<ProbeSecondReport>, | |||
| pub window_boundary: Option<WindowBoundaryReport>, | |||
| pub inference: Option<LimiterInferenceReport>, | |||
| } | |||
| ``` | |||
| Use serde rename attributes so JSON uses snake_case values like `token_bucket`. | |||
| - [ ] **Step 2: Summarize phases** | |||
| Add helpers to group scheduled results by phase and second: | |||
| - burst sent/success/failure | |||
| - refill probe per second | |||
| - sliding probe per second | |||
| - boundary before/after sent/success/failure | |||
| - [ ] **Step 3: Implement diagnose inference** | |||
| Best-effort rules: | |||
| - If boundary after batch succeeds much more than before batch, infer `fixed_window` with medium confidence. | |||
| - If refill probes recover near `rpm / 60` requests per second after burst, infer `token_bucket` with medium confidence. | |||
| - If probes mostly fail until near 60 seconds after burst, infer `sliding_window` with medium confidence. | |||
| - Otherwise infer `unknown` with low confidence. | |||
| Do not overclaim; include signals in report. | |||
| - [ ] **Step 4: Add report tests** | |||
| Test: | |||
| - RPM report serializes `mode`. | |||
| - RPM report does not serialize raw prompt. | |||
| - Token bucket report includes burst and refill probe fields. | |||
| - Diagnose inference serializes expected snake_case limiter value. | |||
| - [ ] **Step 5: Verify and commit** | |||
| Run: | |||
| ```bash | |||
| cargo fmt | |||
| cargo test report rpm_modes | |||
| cargo test | |||
| ``` | |||
| Commit: | |||
| ```bash | |||
| git add src/report.rs src/cli.rs src/rpm_modes.rs | |||
| git commit -m "feat: report rpm limiter mode details" | |||
| ``` | |||
| ## Task 4: Documentation And Final Verification | |||
| **Files:** | |||
| - Modify: `README.md` | |||
| - [ ] **Step 1: Document six modes** | |||
| Add a section for: | |||
| - `sustained`: stable RPM. | |||
| - `burst`: instant burst capacity. | |||
| - `token-bucket`: burst plus refill probe. | |||
| - `sliding-window`: rolling 60 second recovery probe. | |||
| - `window-boundary`: fixed minute boundary probe. | |||
| - `diagnose`: combined unknown-mode diagnosis. | |||
| - [ ] **Step 2: Document interpretation caveats** | |||
| Mention that real LLM backends often combine RPM, TPM, concurrency, account-level, model-level, and region-level limits. Reports are strong signals, not perfect proof. | |||
| - [ ] **Step 3: Final verification** | |||
| Run: | |||
| ```bash | |||
| cargo fmt --check | |||
| cargo test | |||
| cargo clippy --all-targets -- -D warnings | |||
| cargo run -- rpm --help | |||
| ``` | |||
| Do not run live network tests unless the user explicitly asks. | |||
| - [ ] **Step 4: Commit** | |||
| ```bash | |||
| git add README.md | |||
| git commit -m "docs: document rpm limiter modes" | |||
| ``` | |||
| ## Self-Review | |||
| - Spec coverage: covers all six requested modes and keeps existing sustained behavior compatible. | |||
| - Document scan: no unresolved planning gaps. | |||
| - Security: reports must not store prompts or tokens. | |||
| - Testability: schedule, parser, report, and inference tests avoid network calls. | |||