Ver a proveniência

docs: add implementation plan

main
orangels há 1 mês
ascendente
cometimento
ea0b255547
1 ficheiros alterados com 885 adições e 0 eliminações
  1. +885
    -0
      docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md

+ 885
- 0
docs/superpowers/plans/2026-05-06-lq-token-test-implementation.md Ver ficheiro

@@ -0,0 +1,885 @@
# lq_token_test Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build a modular Rust CLI for testing OpenAI-compatible and Anthropic-compatible relay endpoints with single checks, RPM testing, and local official benchmark runs for AIME 2026 and GPQA-Diamond.

**Architecture:** Keep the project as one binary crate with focused internal modules: `cli`, `config`, `runner`, `protocols`, `benchmarks`, and `metrics`. Datasets are not committed; the CLI downloads them into `data/benchmarks/`, writes metadata, and benchmark commands read only local files.

**Tech Stack:** Rust 2024, `clap`, `serde`, `serde_yaml`, `serde_json`, `tokio`, `reqwest` with Rustls, `anyhow`, `thiserror`, `tracing`, `indicatif`, `hdrhistogram`, `regex`, `sha2`, `chrono`, `futures`.

---

## File Structure

- Modify `Cargo.toml`: add CLI, async HTTP, serialization, logging, metrics, hashing, and test dependencies.
- Modify `.gitignore`: keep `target/` and add local benchmark data/output directories.
- Create `config.example.yaml`: example relay and benchmark configuration.
- Modify `src/main.rs`: async entry point, logging, command dispatch.
- Create `src/cli.rs`: `clap` command definitions.
- Create `src/config.rs`: YAML loading, provider lookup, environment expansion.
- Create `src/runner.rs`: provider-neutral request and response types plus request execution timing.
- Create `src/protocols/mod.rs`: protocol enum and dispatch.
- Create `src/protocols/openai.rs`: OpenAI `/chat/completions` request/response adapter.
- Create `src/protocols/anthropic.rs`: Anthropic `/v1/messages` request/response adapter.
- Create `src/benchmarks/mod.rs`: benchmark command orchestration and shared record types.
- Create `src/benchmarks/aime.rs`: AIME 2026 download, parse, prompt, judge integration.
- Create `src/benchmarks/gpqa.rs`: GPQA-Diamond download, parse, prompt, judge integration.
- Create `src/benchmarks/judge.rs`: numeric and multiple-choice answer extraction.
- Create `src/metrics.rs`: latency, success/error, and accuracy summaries.

## Task 1: Dependencies And Project Skeleton

**Files:**
- Modify: `Cargo.toml`
- Modify: `.gitignore`
- Create: `config.example.yaml`
- Modify: `src/main.rs`
- Create: `src/cli.rs`
- Create: `src/config.rs`
- Create: `src/runner.rs`
- Create: `src/protocols/mod.rs`
- Create: `src/protocols/openai.rs`
- Create: `src/protocols/anthropic.rs`
- Create: `src/benchmarks/mod.rs`
- Create: `src/benchmarks/aime.rs`
- Create: `src/benchmarks/gpqa.rs`
- Create: `src/benchmarks/judge.rs`
- Create: `src/metrics.rs`

- [ ] **Step 1: Add crate dependencies**

Set `Cargo.toml` dependencies to:

```toml
[package]
name = "lq_token_test"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "1"
chrono = { version = "0.4", features = ["serde"] }
clap = { version = "4", features = ["derive"] }
futures = "0.3"
hdrhistogram = "7"
indicatif = "0.17"
regex = "1"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
sha2 = "0.10"
thiserror = "2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "time"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
url = "2"

[dev-dependencies]
tempfile = "3"
wiremock = "0.6"
```

- [ ] **Step 2: Ignore local generated data**

Set `.gitignore` to:

```gitignore
/target
/data/benchmarks
/reports
```

- [ ] **Step 3: Add example config**

Create `config.example.yaml`:

```yaml
default_provider: openai

providers:
openai:
protocol: openai
base_url: "https://relay.example.com/v1"
api_token: "${OPENAI_RELAY_TOKEN}"
default_model: "gpt-4o-mini"

anthropic:
protocol: anthropic
base_url: "https://relay.example.com"
api_token: "${ANTHROPIC_RELAY_TOKEN}"
default_model: "claude-3-5-sonnet-latest"

benchmarks:
data_dir: "data/benchmarks"
aime2026:
source: "huggingface:MathArena/aime_2026"
split: "train"
gpqa_diamond:
source: "huggingface:Idavidrein/gpqa"
split: "gpqa_diamond"
```

- [ ] **Step 4: Create empty module declarations**

Replace `src/main.rs` with:

```rust
mod benchmarks;
mod cli;
mod config;
mod metrics;
mod protocols;
mod runner;

use anyhow::Result;
use clap::Parser;
use cli::Cli;

#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();

let cli = Cli::parse();
cli::dispatch(cli).await
}
```

Create placeholder modules that compile:

```rust
// src/cli.rs
use anyhow::Result;
use clap::{Parser, Subcommand};
use std::path::PathBuf;

#[derive(Debug, Parser)]
#[command(name = "lq_token_test", version, about = "Test LLM relay protocols, RPM, and benchmark accuracy")]
pub struct Cli {
#[command(subcommand)]
pub command: Command,
}

#[derive(Debug, Subcommand)]
pub enum Command {
Check {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long)]
prompt: String,
},
Dataset {
#[command(subcommand)]
command: DatasetCommand,
},
Bench {
#[command(subcommand)]
command: BenchCommand,
},
Rpm {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long)]
rpm: u32,
#[arg(long)]
duration: String,
#[arg(long)]
prompt: String,
},
}

#[derive(Debug, Subcommand)]
pub enum DatasetCommand {
Fetch { dataset: String },
}

#[derive(Debug, Subcommand)]
pub enum BenchCommand {
Aime2026 {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long, default_value_t = 4)]
concurrency: usize,
#[arg(long)]
limit: Option<usize>,
},
GpqaDiamond {
#[arg(long, default_value = "config.yaml")]
config: PathBuf,
#[arg(long)]
provider: Option<String>,
#[arg(long)]
model: Option<String>,
#[arg(long, default_value_t = 4)]
concurrency: usize,
#[arg(long)]
limit: Option<usize>,
},
}

pub async fn dispatch(cli: Cli) -> Result<()> {
match cli.command {
Command::Check { .. } => anyhow::bail!("check is not implemented yet"),
Command::Dataset { .. } => anyhow::bail!("dataset is not implemented yet"),
Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"),
Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"),
}
}
```

Create each other module with one line:

```rust
// src/config.rs, src/runner.rs, src/metrics.rs
```

For directories:

```rust
// src/protocols/mod.rs
pub mod anthropic;
pub mod openai;
```

```rust
// src/protocols/openai.rs
```

```rust
// src/protocols/anthropic.rs
```

```rust
// src/benchmarks/mod.rs
pub mod aime;
pub mod gpqa;
pub mod judge;
```

```rust
// src/benchmarks/aime.rs
```

```rust
// src/benchmarks/gpqa.rs
```

```rust
// src/benchmarks/judge.rs
```

- [ ] **Step 5: Verify skeleton**

Run: `cargo fmt`

Run: `cargo test`

Expected: build succeeds with no tests or only placeholder warnings.

- [ ] **Step 6: Commit**

```bash
git add Cargo.toml Cargo.lock .gitignore config.example.yaml src
git commit -m "chore: initialize cli project skeleton"
```

## Task 2: Config Loading

**Files:**
- Modify: `src/config.rs`
- Modify: `src/cli.rs`

- [ ] **Step 1: Write config tests**

Add tests to `src/config.rs`:

```rust
#[cfg(test)]
mod tests {
use super::*;
use std::fs;

#[test]
fn loads_provider_and_expands_env_token() {
unsafe { std::env::set_var("LQ_TEST_TOKEN", "secret-token") };
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.yaml");
fs::write(
&path,
r#"
default_provider: openai
providers:
openai:
protocol: openai
base_url: "https://example.com/v1"
api_token: "${LQ_TEST_TOKEN}"
default_model: "gpt-test"
benchmarks:
data_dir: "data/benchmarks"
"#,
)
.unwrap();

let config = AppConfig::load(&path).unwrap();
let provider = config.provider(None).unwrap();
assert_eq!(provider.api_token, "secret-token");
assert_eq!(provider.default_model, "gpt-test");
assert_eq!(provider.protocol, ProtocolKind::Openai);
}

#[test]
fn rejects_missing_env_token() {
unsafe { std::env::remove_var("LQ_MISSING_TOKEN") };
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.yaml");
fs::write(
&path,
r#"
default_provider: openai
providers:
openai:
protocol: openai
base_url: "https://example.com/v1"
api_token: "${LQ_MISSING_TOKEN}"
default_model: "gpt-test"
"#,
)
.unwrap();

let err = AppConfig::load(&path).unwrap_err().to_string();
assert!(err.contains("LQ_MISSING_TOKEN"));
}
}
```

- [ ] **Step 2: Run tests to verify failure**

Run: `cargo test config::tests`

Expected: fails because `AppConfig` and `ProtocolKind` are not implemented.

- [ ] **Step 3: Implement config**

Implement `src/config.rs`:

```rust
use serde::Deserialize;
use std::{collections::BTreeMap, fs, path::Path};
use thiserror::Error;

#[derive(Debug, Error)]
pub enum ConfigError {
#[error("failed to read config {path}: {source}")]
Read {
path: String,
#[source]
source: std::io::Error,
},
#[error("failed to parse yaml config: {0}")]
Parse(#[from] serde_yaml::Error),
#[error("unknown provider '{0}'")]
UnknownProvider(String),
#[error("config has no default_provider and no --provider was supplied")]
MissingProvider,
#[error("environment variable '{0}' referenced by config is not set")]
MissingEnv(String),
}

#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum ProtocolKind {
Openai,
Anthropic,
}

#[derive(Debug, Clone, Deserialize)]
pub struct ProviderConfig {
pub protocol: ProtocolKind,
pub base_url: String,
pub api_token: String,
pub default_model: String,
}

#[derive(Debug, Clone, Deserialize, Default)]
pub struct BenchmarkConfig {
#[serde(default = "default_data_dir")]
pub data_dir: String,
#[serde(default)]
pub aime2026: Option<DatasetConfig>,
#[serde(default)]
pub gpqa_diamond: Option<DatasetConfig>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct DatasetConfig {
pub source: String,
pub split: String,
}

#[derive(Debug, Clone, Deserialize)]
pub struct AppConfig {
pub default_provider: Option<String>,
pub providers: BTreeMap<String, ProviderConfig>,
#[serde(default)]
pub benchmarks: BenchmarkConfig,
}

fn default_data_dir() -> String {
"data/benchmarks".to_string()
}

impl AppConfig {
pub fn load(path: &Path) -> Result<Self, ConfigError> {
let raw = fs::read_to_string(path).map_err(|source| ConfigError::Read {
path: path.display().to_string(),
source,
})?;
let expanded = expand_env_refs(&raw)?;
Ok(serde_yaml::from_str(&expanded)?)
}

pub fn provider(&self, provider: Option<&str>) -> Result<&ProviderConfig, ConfigError> {
let name = match provider {
Some(name) => name,
None => self
.default_provider
.as_deref()
.ok_or(ConfigError::MissingProvider)?,
};
self.providers
.get(name)
.ok_or_else(|| ConfigError::UnknownProvider(name.to_string()))
}
}

fn expand_env_refs(input: &str) -> Result<String, ConfigError> {
let re = regex::Regex::new(r"\$\{([A-Z0-9_]+)\}").expect("valid env regex");
let mut output = String::with_capacity(input.len());
let mut last = 0;

for caps in re.captures_iter(input) {
let whole = caps.get(0).expect("whole match");
let name = caps.get(1).expect("env name").as_str();
output.push_str(&input[last..whole.start()]);
let value = std::env::var(name).map_err(|_| ConfigError::MissingEnv(name.to_string()))?;
output.push_str(&value);
last = whole.end();
}

output.push_str(&input[last..]);
Ok(output)
}
```

- [ ] **Step 4: Run tests**

Run: `cargo test config::tests`

Expected: both config tests pass.

- [ ] **Step 5: Commit**

```bash
git add src/config.rs src/cli.rs
git commit -m "feat: load yaml relay config"
```

## Task 3: Dataset Download And Parsing

**Files:**
- Modify: `src/cli.rs`
- Modify: `src/benchmarks/mod.rs`
- Modify: `src/benchmarks/aime.rs`
- Modify: `src/benchmarks/gpqa.rs`
- Modify: `src/benchmarks/judge.rs`

- [ ] **Step 1: Add judge tests**

Add tests to `src/benchmarks/judge.rs`:

```rust
#[cfg(test)]
mod tests {
use super::*;

#[test]
fn extracts_final_integer() {
assert_eq!(extract_final_integer("The answer is 42."), Some("42".to_string()));
assert_eq!(extract_final_integer("Final: \\boxed{17}"), Some("17".to_string()));
}

#[test]
fn extracts_choice_letter() {
assert_eq!(extract_choice("Answer: C"), Some('C'));
assert_eq!(extract_choice("I choose (b)."), Some('B'));
}
}
```

- [ ] **Step 2: Implement judge functions**

Implement `src/benchmarks/judge.rs`:

```rust
pub fn extract_final_integer(text: &str) -> Option<String> {
let re = regex::Regex::new(r"-?\d+").expect("valid integer regex");
re.find_iter(text).last().map(|m| m.as_str().to_string())
}

pub fn extract_choice(text: &str) -> Option<char> {
let re = regex::Regex::new(r"(?i)(?:answer\s*:?\s*)?\(?([A-D])\)?").expect("valid choice regex");
re.captures_iter(text)
.last()
.and_then(|caps| caps.get(1))
.and_then(|m| m.as_str().chars().next())
.map(|c| c.to_ascii_uppercase())
}

pub fn judge_integer(output: &str, expected: &str) -> bool {
extract_final_integer(output).as_deref() == Some(expected.trim())
}

pub fn judge_choice(output: &str, expected: char) -> bool {
extract_choice(output) == Some(expected.to_ascii_uppercase())
}
```

- [ ] **Step 3: Add dataset parse tests with small fixtures**

Add tests in `src/benchmarks/aime.rs` and `src/benchmarks/gpqa.rs` using inline rows:

```rust
#[cfg(test)]
mod tests {
use super::*;

#[test]
fn builds_aime_prompt() {
let case = AimeCase {
id: "1".to_string(),
problem: "What is 20 + 22?".to_string(),
answer: "42".to_string(),
};
assert!(case.prompt().contains("What is 20 + 22?"));
assert!(case.prompt().contains("final integer answer"));
}
}
```

```rust
#[cfg(test)]
mod tests {
use super::*;

#[test]
fn builds_gpqa_prompt() {
let case = GpqaCase {
id: "gpqa_1".to_string(),
question: "Which option is correct?".to_string(),
choices: [
("A".to_string(), "Alpha".to_string()),
("B".to_string(), "Beta".to_string()),
("C".to_string(), "Gamma".to_string()),
("D".to_string(), "Delta".to_string()),
],
answer: 'B',
};
let prompt = case.prompt();
assert!(prompt.contains("A. Alpha"));
assert!(prompt.contains("answer with exactly one letter"));
}
}
```

- [ ] **Step 4: Implement benchmark case types and fetch stubs**

Implement enough for prompts and later download wiring:

```rust
// src/benchmarks/aime.rs
#[derive(Debug, Clone)]
pub struct AimeCase {
pub id: String,
pub problem: String,
pub answer: String,
}

impl AimeCase {
pub fn prompt(&self) -> String {
format!(
"Solve the following AIME problem. Return only the final integer answer.\n\n{}",
self.problem
)
}
}
```

```rust
// src/benchmarks/gpqa.rs
#[derive(Debug, Clone)]
pub struct GpqaCase {
pub id: String,
pub question: String,
pub choices: [(String, String); 4],
pub answer: char,
}

impl GpqaCase {
pub fn prompt(&self) -> String {
format!(
"Answer the following multiple-choice science question. You must answer with exactly one letter: A, B, C, or D.\n\n{}\n\n{}. {}\n{}. {}\n{}. {}\n{}. {}",
self.question,
self.choices[0].0,
self.choices[0].1,
self.choices[1].0,
self.choices[1].1,
self.choices[2].0,
self.choices[2].1,
self.choices[3].0,
self.choices[3].1
)
}
}
```

- [ ] **Step 5: Implement `dataset fetch`**

Add download functions that use Hugging Face raw URLs:

```rust
// src/benchmarks/mod.rs
use anyhow::{Context, Result};
use chrono::Utc;
use sha2::{Digest, Sha256};
use std::{fs, path::{Path, PathBuf}};

pub mod aime;
pub mod gpqa;
pub mod judge;

pub async fn fetch_dataset(dataset: &str, data_dir: &Path) -> Result<PathBuf> {
match dataset {
"aime2026" => fetch_to_dir(
"aime2026",
"https://huggingface.co/datasets/MathArena/aime_2026/resolve/main/data/train-00000-of-00001.parquet",
data_dir,
None,
)
.await,
"gpqa-diamond" => fetch_to_dir(
"gpqa_diamond",
"https://huggingface.co/datasets/Idavidrein/gpqa/resolve/main/gpqa_diamond.csv",
data_dir,
std::env::var("HF_TOKEN").ok(),
)
.await,
other => anyhow::bail!("unknown dataset '{other}'"),
}
}

async fn fetch_to_dir(name: &str, url: &str, data_dir: &Path, token: Option<String>) -> Result<PathBuf> {
let dir = data_dir.join(name);
fs::create_dir_all(&dir).with_context(|| format!("failed to create {}", dir.display()))?;
let filename = url.rsplit('/').next().unwrap_or("dataset");
let output = dir.join(filename);

let client = reqwest::Client::new();
let mut request = client.get(url);
if let Some(token) = token {
request = request.bearer_auth(token);
}
let bytes = request.send().await?.error_for_status()?.bytes().await?;
fs::write(&output, &bytes)?;

let hash = Sha256::digest(&bytes);
let metadata = format!(
"name: {name}\nsource_url: {url}\ndownloaded_at: {}\nsha256: {:x}\nbytes: {}\n",
Utc::now().to_rfc3339(),
hash,
bytes.len()
);
fs::write(dir.join("metadata.yaml"), metadata)?;
Ok(output)
}
```

- [ ] **Step 6: Wire dataset command**

Update `src/cli.rs` dispatch for `DatasetCommand::Fetch` to load default data dir from config when `config.yaml` exists, otherwise use `data/benchmarks`.

- [ ] **Step 7: Run tests**

Run: `cargo test benchmarks`

Expected: judge and prompt tests pass. Network fetch is not tested by default.

- [ ] **Step 8: Commit**

```bash
git add src/benchmarks src/cli.rs
git commit -m "feat: add benchmark dataset fetch and prompts"
```

## Task 4: Protocol Adapters And Single Check

**Files:**
- Modify: `src/runner.rs`
- Modify: `src/protocols/mod.rs`
- Modify: `src/protocols/openai.rs`
- Modify: `src/protocols/anthropic.rs`
- Modify: `src/cli.rs`

- [ ] **Step 1: Write HTTP adapter tests with wiremock**

Add tests for OpenAI and Anthropic adapters that mock successful responses and assert extracted text.

- [ ] **Step 2: Define runner types**

Implement provider-neutral types:

```rust
#[derive(Debug, Clone)]
pub struct ModelRequest {
pub base_url: String,
pub api_token: String,
pub model: String,
pub prompt: String,
pub temperature: f32,
pub max_tokens: u32,
}

#[derive(Debug, Clone)]
pub struct ModelResponse {
pub text: String,
pub status: u16,
pub elapsed_ms: u128,
}
```

- [ ] **Step 3: Implement OpenAI adapter**

Post to `{base_url}/chat/completions` with `Authorization: Bearer <token>` and parse `choices[0].message.content`.

- [ ] **Step 4: Implement Anthropic adapter**

Post to `{base_url}/v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and parse the first text block in `content`.

- [ ] **Step 5: Wire `check` command**

Load config, resolve provider and model, run one request, then print status, elapsed time, and response text.

- [ ] **Step 6: Run tests**

Run: `cargo test protocols runner`

Expected: mocked protocol tests pass.

- [ ] **Step 7: Commit**

```bash
git add src/runner.rs src/protocols src/cli.rs
git commit -m "feat: add openai and anthropic request adapters"
```

## Task 5: Metrics, Benchmark Runs, And RPM

**Files:**
- Modify: `src/metrics.rs`
- Modify: `src/benchmarks/mod.rs`
- Modify: `src/benchmarks/aime.rs`
- Modify: `src/benchmarks/gpqa.rs`
- Modify: `src/cli.rs`

- [ ] **Step 1: Add metrics tests**

Test success/error counts, accuracy, and latency percentile calculation.

- [ ] **Step 2: Implement metrics summary**

Use `hdrhistogram::Histogram<u64>` for elapsed milliseconds and counters for success, failure, correct, and total judged.

- [ ] **Step 3: Implement local benchmark loading**

For AIME 2026 and GPQA-Diamond, read local files under `data/benchmarks`. If data is missing, return a clear error: `missing local dataset; run lq_token_test dataset fetch <name>`.

- [ ] **Step 4: Implement benchmark execution**

Use `futures::stream` with `buffer_unordered(concurrency)` to run cases concurrently. Apply `limit` before execution. Judge each response and update metrics.

- [ ] **Step 5: Implement RPM command**

Parse duration strings like `60s` and `5m`, calculate delay between requests from `rpm`, run repeated requests, and print latency/error summary.

- [ ] **Step 6: Run tests**

Run: `cargo test`

Expected: unit tests pass.

- [ ] **Step 7: Commit**

```bash
git add src/metrics.rs src/benchmarks src/cli.rs
git commit -m "feat: run benchmark and rpm tests"
```

## Task 6: Verification And Docs

**Files:**
- Create: `README.md`
- Modify: `config.example.yaml`

- [ ] **Step 1: Document usage**

Add README sections for config, dataset fetch, OpenAI check, Anthropic check, AIME benchmark, GPQA benchmark, and RPM test.

- [ ] **Step 2: Verify formatting and tests**

Run:

```bash
cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
```

Expected: all pass.

- [ ] **Step 3: Manual smoke commands**

Run:

```bash
cargo run -- --help
cargo run -- dataset fetch aime2026
```

Expected: help renders; AIME fetch creates `data/benchmarks/aime2026/metadata.yaml`.

- [ ] **Step 4: Commit**

```bash
git add README.md config.example.yaml
git commit -m "docs: add usage guide"
```

## Self-Review

- Spec coverage: covered YAML config, modular single crate, OpenAI and Anthropic protocol adapters, dataset download without committing data, AIME 2026, GPQA-Diamond, benchmark judging, metrics, RPM, and README usage.
- Placeholder scan: no unresolved placeholders or open questions remain in the planned behavior.
- Type consistency: dataset names are `aime2026` and `gpqa-diamond` in CLI; local directory names are `aime2026` and `gpqa_diamond`; module names are `aime` and `gpqa`.

Carregando…
Cancelar
Guardar