api模型检测
25개 이상의 토픽을 선택하실 수 없습니다. Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

112 lines
2.8 KiB

  1. use crate::benchmarks;
  2. use crate::config::AppConfig;
  3. use anyhow::Result;
  4. use clap::{Parser, Subcommand};
  5. use std::path::{Path, PathBuf};
  6. #[derive(Debug, Parser)]
  7. #[command(
  8. name = "lq_token_test",
  9. version,
  10. about = "Test LLM relay protocols, RPM, and benchmark accuracy"
  11. )]
  12. pub struct Cli {
  13. #[command(subcommand)]
  14. pub command: Command,
  15. }
  16. #[derive(Debug, Subcommand)]
  17. pub enum Command {
  18. Check {
  19. #[arg(long, default_value = "config.yaml")]
  20. config: PathBuf,
  21. #[arg(long)]
  22. provider: Option<String>,
  23. #[arg(long)]
  24. model: Option<String>,
  25. #[arg(long)]
  26. prompt: String,
  27. },
  28. Dataset {
  29. #[command(subcommand)]
  30. command: DatasetCommand,
  31. },
  32. Bench {
  33. #[command(subcommand)]
  34. command: BenchCommand,
  35. },
  36. Rpm {
  37. #[arg(long, default_value = "config.yaml")]
  38. config: PathBuf,
  39. #[arg(long)]
  40. provider: Option<String>,
  41. #[arg(long)]
  42. model: Option<String>,
  43. #[arg(long)]
  44. rpm: u32,
  45. #[arg(long)]
  46. duration: String,
  47. #[arg(long)]
  48. prompt: String,
  49. },
  50. }
  51. #[derive(Debug, Subcommand)]
  52. pub enum DatasetCommand {
  53. Fetch { dataset: String },
  54. }
  55. #[derive(Debug, Subcommand)]
  56. pub enum BenchCommand {
  57. Aime2026 {
  58. #[arg(long, default_value = "config.yaml")]
  59. config: PathBuf,
  60. #[arg(long)]
  61. provider: Option<String>,
  62. #[arg(long)]
  63. model: Option<String>,
  64. #[arg(long, default_value_t = 4)]
  65. concurrency: usize,
  66. #[arg(long)]
  67. limit: Option<usize>,
  68. },
  69. GpqaDiamond {
  70. #[arg(long, default_value = "config.yaml")]
  71. config: PathBuf,
  72. #[arg(long)]
  73. provider: Option<String>,
  74. #[arg(long)]
  75. model: Option<String>,
  76. #[arg(long, default_value_t = 4)]
  77. concurrency: usize,
  78. #[arg(long)]
  79. limit: Option<usize>,
  80. },
  81. }
  82. pub async fn dispatch(cli: Cli) -> Result<()> {
  83. match cli.command {
  84. Command::Check { .. } => anyhow::bail!("check is not implemented yet"),
  85. Command::Dataset {
  86. command: DatasetCommand::Fetch { dataset },
  87. } => {
  88. let data_dir = dataset_data_dir(Path::new("config.yaml"));
  89. let path = benchmarks::fetch_dataset(&dataset, &data_dir).await?;
  90. println!("{}", path.display());
  91. Ok(())
  92. }
  93. Command::Bench { .. } => anyhow::bail!("bench is not implemented yet"),
  94. Command::Rpm { .. } => anyhow::bail!("rpm is not implemented yet"),
  95. }
  96. }
  97. fn dataset_data_dir(config_path: &Path) -> PathBuf {
  98. if config_path.exists() {
  99. if let Ok(config) = AppConfig::load(config_path) {
  100. return PathBuf::from(config.benchmarks.data_dir);
  101. }
  102. }
  103. PathBuf::from("data/benchmarks")
  104. }