| @@ -0,0 +1,398 @@ | |||||
| # RPM Limiter Modes Implementation Plan | |||||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | |||||
| **Goal:** Extend `rpm` testing from one sustained schedule into six explicit limiter test modes: `sustained`, `burst`, `token-bucket`, `sliding-window`, `window-boundary`, and `diagnose`. | |||||
| **Architecture:** Keep mode-specific scheduling and inference in a focused module, while `cli` remains responsible for argument parsing and command dispatch. Reports should preserve the current RPM summary shape and add mode-specific detail without storing raw prompts or secrets. | |||||
| **Tech Stack:** Rust 2024, existing `clap`, `tokio`, `futures`, `serde`, `chrono`, `hdrhistogram`, `anyhow`. | |||||
| --- | |||||
| ## File Structure | |||||
| - Modify `src/cli.rs`: add `--mode`, mode-specific flags, dispatch to mode runner, and keep current sustained behavior as default. | |||||
| - Create `src/rpm_modes.rs`: mode enum, schedules, probe plans, limiter inference, and unit tests. | |||||
| - Modify `src/report.rs`: extend RPM reports with mode, optional burst/probe/window/diagnose details, actual RPM, and inferred limiter. | |||||
| - Modify `src/main.rs`: register `rpm_modes` module. | |||||
| - Modify `README.md`: document RPM modes and example commands. | |||||
| ## Mode Semantics | |||||
| - `sustained`: current default. Starts one request every `60 / rpm` seconds for `duration`. Requests may overlap. | |||||
| - `burst`: starts `burst` requests at `t=0` and reports immediate success/failure/latency/error behavior. | |||||
| - `token-bucket`: starts `burst` requests at `t=0`, then probes refill behavior for `probe_seconds`. Probe rate defaults to the expected refill rate from `rpm`, rounded up to at least one probe per second. | |||||
| - `sliding-window`: starts `burst` requests at `t=0`, then sends low-rate probes for `probe_seconds` to observe whether recovery happens near the rolling 60 second boundary. | |||||
| - `window-boundary`: waits until the next minute boundary test point, sends one batch before the boundary and one batch after it. This tests fixed-window reset behavior. | |||||
| - `diagnose`: runs a bounded combined probe using burst, refill probes, and boundary signals, then writes a best-effort inference: `token_bucket`, `fixed_window`, `sliding_window`, or `unknown`. | |||||
| ## CLI Shape | |||||
| Keep the old command valid: | |||||
| ```bash | |||||
| cargo run -- rpm --provider anthropic --rpm 120 --duration 60s --prompt "hello" | |||||
| ``` | |||||
| Equivalent explicit command: | |||||
| ```bash | |||||
| cargo run -- rpm --mode sustained --provider anthropic --rpm 120 --duration 60s --prompt "hello" | |||||
| ``` | |||||
| New commands: | |||||
| ```bash | |||||
| cargo run -- rpm --mode burst --provider anthropic --burst 120 --prompt "hello" | |||||
| cargo run -- rpm --mode token-bucket --provider anthropic --rpm 120 --burst 120 --probe-seconds 30 --prompt "hello" | |||||
| cargo run -- rpm --mode sliding-window --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" | |||||
| cargo run -- rpm --mode window-boundary --provider anthropic --rpm 120 --burst 120 --prompt "hello" | |||||
| cargo run -- rpm --mode diagnose --provider anthropic --rpm 120 --burst 120 --probe-seconds 90 --prompt "hello" | |||||
| ``` | |||||
| Argument defaults: | |||||
| - `--mode sustained` | |||||
| - `--duration 60s`, required only by `sustained` | |||||
| - `--burst <rpm>` default for burst-style modes when omitted | |||||
| - `--probe-seconds 90` for `sliding-window` and `diagnose` | |||||
| - `--probe-seconds 30` for `token-bucket` | |||||
| - `--window-offset-ms 500` for `window-boundary`, meaning send before/after batches around the next minute boundary | |||||
| - `--concurrency` should cap in-flight requests for all modes; default can be `burst` or a safe high value, but must never be zero | |||||
| ## Report Shape | |||||
| Extend `RpmReport` with: | |||||
| ```json | |||||
| { | |||||
| "mode": "token-bucket", | |||||
| "provider": "anthropic", | |||||
| "model": "claude-test", | |||||
| "run": { | |||||
| "started_at": "2026-05-06T15:30:12Z", | |||||
| "duration_ms": 60000, | |||||
| "target_rpm": 120, | |||||
| "actual_rpm": 118.5, | |||||
| "temperature": 0.0, | |||||
| "max_tokens": 1024 | |||||
| }, | |||||
| "summary": { | |||||
| "actual_requests": 120, | |||||
| "success": 118, | |||||
| "failure": 2, | |||||
| "latency_ms": { | |||||
| "p50": 800, | |||||
| "p95": 1500, | |||||
| "p99": 2200 | |||||
| } | |||||
| }, | |||||
| "mode_detail": { | |||||
| "burst": { | |||||
| "sent": 120, | |||||
| "success": 118, | |||||
| "failure": 2 | |||||
| }, | |||||
| "refill_probe": [ | |||||
| { | |||||
| "second": 1, | |||||
| "sent": 2, | |||||
| "success": 2, | |||||
| "failure": 0 | |||||
| } | |||||
| ], | |||||
| "inference": { | |||||
| "likely_limiter": "token_bucket", | |||||
| "confidence": "medium", | |||||
| "signals": [ | |||||
| "burst accepted most initial requests", | |||||
| "probe success approximated 2.00 req/s refill" | |||||
| ] | |||||
| } | |||||
| }, | |||||
| "errors": [] | |||||
| } | |||||
| ``` | |||||
| Rules: | |||||
| - Do not serialize raw prompt. | |||||
| - Do not serialize tokens. | |||||
| - Include `mode` at top level. | |||||
| - Include `actual_rpm` for every mode where duration is meaningful. | |||||
| - Include mode detail only when relevant. | |||||
| - Existing sustained reports remain easy to read. | |||||
| ## Task 1: Mode Types And Scheduling | |||||
| **Files:** | |||||
| - Create: `src/rpm_modes.rs` | |||||
| - Modify: `src/main.rs` | |||||
| - [ ] **Step 1: Add mode enum and parser** | |||||
| Implement `RpmMode` with `clap::ValueEnum`: | |||||
| ```rust | |||||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum)] | |||||
| #[value(rename_all = "kebab-case")] | |||||
| pub enum RpmMode { | |||||
| Sustained, | |||||
| Burst, | |||||
| TokenBucket, | |||||
| SlidingWindow, | |||||
| WindowBoundary, | |||||
| Diagnose, | |||||
| } | |||||
| ``` | |||||
| - [ ] **Step 2: Add scheduling helpers** | |||||
| Implement: | |||||
| ```rust | |||||
| pub fn sustained_schedule(duration: Duration, rpm: u32) -> Vec<Duration>; | |||||
| pub fn burst_schedule(burst: u32) -> Vec<Duration>; | |||||
| pub fn token_bucket_schedule(rpm: u32, burst: u32, probe_seconds: u64) -> Vec<ScheduledProbe>; | |||||
| pub fn sliding_window_schedule(burst: u32, probe_seconds: u64) -> Vec<ScheduledProbe>; | |||||
| pub fn window_boundary_plan(now: DateTime<Utc>, burst: u32, offset_ms: u64) -> WindowBoundaryPlan; | |||||
| ``` | |||||
| Use explicit structs for `ScheduledProbe` and `WindowBoundaryPlan`; include phase labels such as `burst`, `refill_probe`, `sliding_probe`, `before_boundary`, and `after_boundary`. | |||||
| - [ ] **Step 3: Add schedule tests** | |||||
| Test: | |||||
| - 120 RPM sustained creates starts at 0ms, 500ms, 1000ms for a short duration. | |||||
| - Burst 5 creates five 0ms starts. | |||||
| - Token bucket 120 RPM with 120 burst and 2 probe seconds has 120 burst starts plus about 4 refill probes. | |||||
| - Sliding window probe lasts through the requested probe seconds. | |||||
| - Window boundary plan places before and after batches around a minute boundary. | |||||
| - [ ] **Step 4: Register module** | |||||
| Add `mod rpm_modes;` to `src/main.rs`. | |||||
| - [ ] **Step 5: Verify and commit** | |||||
| Run: | |||||
| ```bash | |||||
| cargo fmt | |||||
| cargo test rpm_modes | |||||
| ``` | |||||
| Commit: | |||||
| ```bash | |||||
| git add src/rpm_modes.rs src/main.rs | |||||
| git commit -m "feat: add rpm mode schedules" | |||||
| ``` | |||||
| ## Task 2: CLI Wiring And Mode Execution | |||||
| **Files:** | |||||
| - Modify: `src/cli.rs` | |||||
| - Modify: `src/rpm_modes.rs` | |||||
| - [ ] **Step 1: Extend CLI args** | |||||
| Add to `Command::Rpm`: | |||||
| ```rust | |||||
| #[arg(long, value_enum, default_value_t = RpmMode::Sustained)] | |||||
| mode: RpmMode, | |||||
| #[arg(long)] | |||||
| burst: Option<u32>, | |||||
| #[arg(long)] | |||||
| probe_seconds: Option<u64>, | |||||
| #[arg(long, default_value_t = 500)] | |||||
| window_offset_ms: u64, | |||||
| #[arg(long)] | |||||
| concurrency: Option<usize>, | |||||
| ``` | |||||
| Keep existing `rpm`, `duration`, and `prompt` arguments compatible. `duration` may remain a string with default `60s`; non-sustained modes can ignore it unless needed. | |||||
| - [ ] **Step 2: Build a request runner helper** | |||||
| Refactor RPM execution so all modes share: | |||||
| ```rust | |||||
| async fn run_scheduled_requests( | |||||
| protocol: ProtocolKind, | |||||
| request: ModelRequest, | |||||
| starts: Vec<ScheduledRequest>, | |||||
| max_in_flight: usize, | |||||
| ) -> Vec<ScheduledResult>; | |||||
| ``` | |||||
| Each result should retain phase/second metadata for mode-detail reporting. | |||||
| - [ ] **Step 3: Implement mode execution** | |||||
| Mode behavior: | |||||
| - `sustained`: use current behavior with `sustained_schedule`. | |||||
| - `burst`: use `burst_schedule`. | |||||
| - `token-bucket`: burst at 0, then refill probes based on `rpm`. | |||||
| - `sliding-window`: burst at 0, then one probe per second until `probe_seconds`. | |||||
| - `window-boundary`: sleep until planned before-boundary start, send before batch, then after batch. | |||||
| - `diagnose`: run bounded burst/refill/boundary probes and produce inference. | |||||
| - [ ] **Step 4: Add CLI behavior tests** | |||||
| Add tests for: | |||||
| - Old RPM command defaults to `sustained`. | |||||
| - `--mode token-bucket` parses. | |||||
| - `--mode window-boundary --window-offset-ms 250` parses. | |||||
| - Invalid zero `--burst` and zero `--rpm` are rejected by execution validation. | |||||
| - [ ] **Step 5: Verify and commit** | |||||
| Run: | |||||
| ```bash | |||||
| cargo fmt | |||||
| cargo test cli::tests rpm_modes | |||||
| cargo test | |||||
| ``` | |||||
| Commit: | |||||
| ```bash | |||||
| git add src/cli.rs src/rpm_modes.rs | |||||
| git commit -m "feat: run rpm limiter modes" | |||||
| ``` | |||||
| ## Task 3: Report Extensions And Inference | |||||
| **Files:** | |||||
| - Modify: `src/report.rs` | |||||
| - Modify: `src/cli.rs` | |||||
| - Modify: `src/rpm_modes.rs` | |||||
| - [ ] **Step 1: Extend report structs** | |||||
| Add: | |||||
| ```rust | |||||
| pub enum LimiterInferenceKind { | |||||
| TokenBucket, | |||||
| FixedWindow, | |||||
| SlidingWindow, | |||||
| Unknown, | |||||
| } | |||||
| pub struct LimiterInferenceReport { | |||||
| pub likely_limiter: LimiterInferenceKind, | |||||
| pub confidence: String, | |||||
| pub signals: Vec<String>, | |||||
| } | |||||
| pub struct RpmModeDetailReport { | |||||
| pub burst: Option<PhaseSummaryReport>, | |||||
| pub refill_probe: Vec<ProbeSecondReport>, | |||||
| pub sliding_probe: Vec<ProbeSecondReport>, | |||||
| pub window_boundary: Option<WindowBoundaryReport>, | |||||
| pub inference: Option<LimiterInferenceReport>, | |||||
| } | |||||
| ``` | |||||
| Use serde rename attributes so JSON uses snake_case values like `token_bucket`. | |||||
| - [ ] **Step 2: Summarize phases** | |||||
| Add helpers to group scheduled results by phase and second: | |||||
| - burst sent/success/failure | |||||
| - refill probe per second | |||||
| - sliding probe per second | |||||
| - boundary before/after sent/success/failure | |||||
| - [ ] **Step 3: Implement diagnose inference** | |||||
| Best-effort rules: | |||||
| - If boundary after batch succeeds much more than before batch, infer `fixed_window` with medium confidence. | |||||
| - If refill probes recover near `rpm / 60` requests per second after burst, infer `token_bucket` with medium confidence. | |||||
| - If probes mostly fail until near 60 seconds after burst, infer `sliding_window` with medium confidence. | |||||
| - Otherwise infer `unknown` with low confidence. | |||||
| Do not overclaim; include signals in report. | |||||
| - [ ] **Step 4: Add report tests** | |||||
| Test: | |||||
| - RPM report serializes `mode`. | |||||
| - RPM report does not serialize raw prompt. | |||||
| - Token bucket report includes burst and refill probe fields. | |||||
| - Diagnose inference serializes expected snake_case limiter value. | |||||
| - [ ] **Step 5: Verify and commit** | |||||
| Run: | |||||
| ```bash | |||||
| cargo fmt | |||||
| cargo test report rpm_modes | |||||
| cargo test | |||||
| ``` | |||||
| Commit: | |||||
| ```bash | |||||
| git add src/report.rs src/cli.rs src/rpm_modes.rs | |||||
| git commit -m "feat: report rpm limiter mode details" | |||||
| ``` | |||||
| ## Task 4: Documentation And Final Verification | |||||
| **Files:** | |||||
| - Modify: `README.md` | |||||
| - [ ] **Step 1: Document six modes** | |||||
| Add a section for: | |||||
| - `sustained`: stable RPM. | |||||
| - `burst`: instant burst capacity. | |||||
| - `token-bucket`: burst plus refill probe. | |||||
| - `sliding-window`: rolling 60 second recovery probe. | |||||
| - `window-boundary`: fixed minute boundary probe. | |||||
| - `diagnose`: combined unknown-mode diagnosis. | |||||
| - [ ] **Step 2: Document interpretation caveats** | |||||
| Mention that real LLM backends often combine RPM, TPM, concurrency, account-level, model-level, and region-level limits. Reports are strong signals, not perfect proof. | |||||
| - [ ] **Step 3: Final verification** | |||||
| Run: | |||||
| ```bash | |||||
| cargo fmt --check | |||||
| cargo test | |||||
| cargo clippy --all-targets -- -D warnings | |||||
| cargo run -- rpm --help | |||||
| ``` | |||||
| Do not run live network tests unless the user explicitly asks. | |||||
| - [ ] **Step 4: Commit** | |||||
| ```bash | |||||
| git add README.md | |||||
| git commit -m "docs: document rpm limiter modes" | |||||
| ``` | |||||
| ## Self-Review | |||||
| - Spec coverage: covers all six requested modes and keeps existing sustained behavior compatible. | |||||
| - Document scan: no unresolved planning gaps. | |||||
| - Security: reports must not store prompts or tokens. | |||||
| - Testability: schedule, parser, report, and inference tests avoid network calls. | |||||