fengsilin
3 weeks ago
1 changed files with 166 additions and 0 deletions
Unified View
Diff Options
+ 166
- 0
docs/superpowers/specs/2026-06-18-e2e-test-foundation-design.md
View File
| @@ -0,0 +1,166 @@ | |||||
| # E2E Test Foundation Design | |||||
| ## Goal | |||||
| Build a reusable end-to-end test foundation for the admin console, then use the model multidimensional pricing page as the first covered workflow. | |||||
| The foundation must verify the real frontend/backend contract, authentication, persistence, and page routing. It should be easy to run locally and suitable for CI later. | |||||
| ## Scope | |||||
| In scope: | |||||
| - Add a Playwright-based E2E test foundation under `web/`. | |||||
| - Run the React frontend through Vite. | |||||
| - Run the Go backend against an isolated SQLite database. | |||||
| - Log in as the generated root user. | |||||
| - Cover the model multidimensional pricing workflow under `/console/setting?tab=ratio`. | |||||
| - Keep tests deterministic by using temporary files, fixed ports, and isolated storage state. | |||||
| Out of scope for the first pass: | |||||
| - Docker Compose E2E. | |||||
| - MySQL/PostgreSQL E2E matrix. | |||||
| - Full relay calls to upstream AI providers. | |||||
| - Broad admin console coverage outside the model pricing workflow. | |||||
| ## Recommended Approach | |||||
| Use real backend + temporary SQLite + Playwright. | |||||
| The project already has `@playwright/test` in `web/package.json`. The backend can start with `go run main.go`, and when no users exist it creates the root user with username `root` and password `123456`. Running against a temporary SQLite database gives each test run a clean system without requiring external services. | |||||
| This approach is slower than API mocks, but it catches the issues that matter for this feature: frontend/backend response shape, save/load behavior, auth redirects, and persistence across page reloads. | |||||
| ## Runtime Architecture | |||||
| Playwright will run from the `web/` directory. | |||||
| Backend: | |||||
| - Started by a Playwright global setup helper. | |||||
| - Command: `go run main.go` from the repository root. | |||||
| - Environment: | |||||
| - `PORT=3001` | |||||
| - `SQLITE_PATH=<temp-dir>/e2e.db?_busy_timeout=30000` | |||||
| - `SESSION_SECRET=e2e_session_secret_for_local_tests` | |||||
| - The helper waits for `/api/setup` or another lightweight HTTP endpoint before running tests. | |||||
| - The helper stores backend process metadata so global teardown can stop it. | |||||
| Frontend: | |||||
| - Started by Playwright `webServer`. | |||||
| - Command: `bun run dev -- --host 127.0.0.1 --port 5173` | |||||
| - Environment: | |||||
| - `VITE_REACT_APP_SERVER_URL=http://127.0.0.1:3001` | |||||
| - Base URL: `http://127.0.0.1:5173` | |||||
| Browser auth: | |||||
| - A setup project logs in once as `root / 123456`. | |||||
| - The authenticated browser state is saved to `web/e2e/.auth/root.json`. | |||||
| - Feature tests reuse that storage state. | |||||
| ## Test Layout | |||||
| Proposed files: | |||||
| - `web/playwright.config.js` | |||||
| - `web/e2e/global-setup.js` | |||||
| - `web/e2e/global-teardown.js` | |||||
| - `web/e2e/auth.setup.js` | |||||
| - `web/e2e/model-pricing.spec.js` | |||||
| - `web/e2e/utils/process.js` | |||||
| - `web/e2e/utils/wait.js` | |||||
| Proposed package scripts: | |||||
| - `e2e`: `playwright test` | |||||
| - `e2e:ui`: `playwright test --ui` | |||||
| - `e2e:headed`: `playwright test --headed` | |||||
| - `e2e:install`: `playwright install chromium` | |||||
| ## First Workflow: Model Multidimensional Pricing | |||||
| The first E2E test should cover the critical user path: | |||||
| 1. Open `/console/setting?tab=ratio`. | |||||
| 2. Confirm the admin setting page loads. | |||||
| 3. Open the `模型多维计费` tab. | |||||
| 4. Create or edit pricing for a deterministic test model name, for example `e2e-video-pricing-model`. | |||||
| 5. Add dimensions such as `duration` and `quality`. | |||||
| 6. Use the generator to produce a linear pricing table. | |||||
| 7. Save the config. | |||||
| 8. Reload the page. | |||||
| 9. Reopen the same model config. | |||||
| 10. Assert that dimensions and generated rows are loaded. | |||||
| The test should also observe the network response for save/load requests and assert successful API envelopes where practical. This makes contract mismatches visible even if the UI shows a generic toast. | |||||
| ## Selectors | |||||
| Prefer stable selectors instead of brittle text-only selectors for controls that are hard to disambiguate. | |||||
| The implementation should add `data-testid` only where current markup cannot be selected reliably: | |||||
| - model pricing tab container | |||||
| - create/edit model pricing button | |||||
| - model name input | |||||
| - dimension editor add row button | |||||
| - generator open/apply buttons | |||||
| - save button | |||||
| Visible text can still be used for high-level navigation when the text is unique and user-facing. | |||||
| ## Data Isolation | |||||
| Each E2E run uses a new temporary SQLite database path. This prevents tests from depending on local developer data and avoids cleanup requirements for most tables. | |||||
| Within the UI, test-created models should use an `e2e-` prefix. If the same database is reused during debugging, tests can safely identify their own records. | |||||
| ## Failure Artifacts | |||||
| Playwright should collect useful artifacts on failure: | |||||
| - screenshot: only on failure | |||||
| - trace: retain on failure | |||||
| - video: off by default for speed | |||||
| These defaults keep normal runs fast while still making UI failures diagnosable. | |||||
| ## CI Path | |||||
| The initial implementation should work locally first. CI can later run: | |||||
| 1. `cd web && bun install` | |||||
| 2. `cd web && bun run e2e:install` | |||||
| 3. `cd web && bun run e2e` | |||||
| No external database or service should be required for the first workflow. | |||||
| ## Risks And Mitigations | |||||
| Backend startup can be slow. | |||||
| - Mitigation: wait on an HTTP endpoint with a clear timeout and useful logs. | |||||
| Ports can be occupied locally. | |||||
| - Mitigation: use fixed defaults first for simplicity, but allow overrides through `E2E_BACKEND_PORT` and `E2E_FRONTEND_PORT`. | |||||
| Chinese UI text may be rendered through i18n and encoding-sensitive files. | |||||
| - Mitigation: prefer `data-testid` for critical controls and keep user-facing text assertions limited to stable labels. | |||||
| Playwright browsers may not be installed. | |||||
| - Mitigation: provide `bun run e2e:install` and document it in script names. | |||||
| ## Acceptance Criteria | |||||
| - `cd web && bun run e2e` starts the backend and frontend automatically. | |||||
| - The run uses an isolated SQLite database. | |||||
| - The test logs in as root without manual steps. | |||||
| - The model multidimensional pricing workflow can save and reload a generated config. | |||||
| - Failing tests produce a trace and screenshot. | |||||
| - Existing unit tests and frontend build remain runnable without depending on E2E setup. | |||||