REQUIRED when modifying any file in packages/playground-ui or packages/playground. Triggers on: React component creation/modification/refactoring, UI changes, new playground features, bug fixes affecting studio UI. Generates Playwright E2E tests that validate PRODUCT BEHAVIOR, not just UI states.
npxskills add mastra-ai/mastra--skill e2e-tests-studioLoading…
CRITICAL: Tests must verify that product features WORK correctly, not just that UI elements render.
Every E2E spec MUST follow the same BDD shape as the MSW tests. In packages/playground, e2e-bdd/test-needs-when-describe enforces this shape.
The structure has exactly three levels:
test.describe = the unit under test (one page or feature per file).test.describe('when …') = exactly ONE precondition. The title MUST start with when.test = exactly ONE observable outcome.import { test, expect } from '@playwright/test';
import { resetStorage } from '../__utils__/reset-storage';
test.describe('Tools list page', () => {
// the unit
test.afterEach(async () => {
await resetStorage();
});
test.describe('when a registered tool is clicked', () => {
// ONE precondition (starts with "when")
test('navigates to that tool detail page', async ({ page }) => {
// ONE outcome
await page.goto('/tools');
await page.locator('text=Get current weather for a location').click();
await expect(page).toHaveURL(/\/tools\/weatherInfo$/);
});
test('shows the tool name as the page heading', async ({ page }) => {
// ONE outcome
await page.goto('/tools');
await page.locator('text=Get current weather for a location').click();
await expect(page.locator('h2')).toHaveText('weatherInfo');
});
});
});
Rules:
test.describe per file naming the unit.test lives inside a test.describe('when …') precondition group. No top-level flat test().test() only where assertions represent distinct outcomes; keep tightly-coupled assertions that prove a single outcome together. Never drop an assertion.beforeEach/afterEach in the narrowest describe scope that needs them.Requires Playwright MCP server. If the browser_navigate tool is unavailable, instruct the user to add it:
claude mcp add playwright -- npx @playwright/mcp@latest
Before writing ANY test, answer these questions:
Document these answers as comments in your test file.
pnpm build:cli
cd packages/playground/e2e/kitchen-sink && pnpm dev
Verify server at http://localhost:4111
| Feature Category | What to Test | Example Assertion |
|---|---|---|
| Agent Configuration | Config changes affect agent behavior | Send message → verify response uses selected model |
| LLM Provider Selection | Selected provider is used in requests | Intercept API call → verify provider in request payload |
| Tool Execution | Tool runs with correct params & returns result | Execute tool → verify output matches expected transformation |
| Workflow Execution | Steps execute in order, data flows between steps | Run workflow → verify each step's output feeds next step |
| Chat/Streaming | Messages persist, context maintained across turns | Multi-turn conversation → verify context awareness |
| MCP Server Tools | Server tools are callable and return data | Call MCP tool → verify response structure and content |
| Memory/Persistence | Data survives page reload | Create item → reload → verify item exists |
| Error Handling | Errors surface correctly to user | Trigger error condition → verify error message + recovery |
import { test, expect, Page } from '@playwright/test';
import { resetStorage } from '../__utils__/reset-storage';
import { selectFixture } from '../__utils__/select-fixture';
import { nanoid } from 'nanoid';
/**
* FEATURE: [Name of feature]
* USER STORY: As a user, I want to [action] so that [outcome]
* BEHAVIOR UNDER TEST: [Specific behavior being validated]
*/
test.describe('[Feature Name] - Behavior Tests', () => {
let page: Page;
test.beforeEach(async ({ browser }) => {
const context = await browser.newContext();
page = await context.newPage();
});
test.afterEach(async () => {
await resetStorage(page);
});
test.describe('when [the single precondition for these outcomes]', () => {
test('[verb describing the single observable outcome]', async () => {
// ARRANGE: Set up preconditions
// - Navigate to the feature
// - Configure any required state
// ACT: Perform the user action that triggers the behavior
// ASSERT: Verify the OUTCOME, not the UI state
// - Check data persistence
// - Verify downstream effects
// - Confirm API calls made correctly
});
});
});
test.describe('when a different LLM provider is selected', () => {
test('uses that provider for agent responses', async () => {
// ARRANGE
await page.goto('/agents/my-agent/chat');
// Intercept API to verify provider
let capturedProvider: string | null = null;
await page.route('**/api/chat', route => {
const body = JSON.parse(route.request().postData() || '{}');
capturedProvider = body.provider;
route.continue();
});
// ACT: Select a different provider
await page.getByTestId('provider-selector').click();
await page.getByRole('option', { name: 'OpenAI' }).click();
// Send a message to trigger the agent
await page.getByTestId('chat-input').fill('Hello');
await page.getByTestId('send-button').click();
// ASSERT: Verify the selected provider was used
await expect.poll(() => capturedProvider).toBe('openai');
});
});
test.describe('when a new agent is created', () => {
test('persists after page reload', async () => {
// ARRANGE
await page.goto('/agents');
const agentName = `Test Agent ${nanoid()}`;
// ACT: Create new agent
await page.getByTestId('create-agent-button').click();
await page.getByTestId('agent-name-input').fill(agentName);
await page.getByTestId('save-agent-button').click();
// Wait for creation to complete
await expect(page.getByText(agentName)).toBeVisible();
// ASSERT: Verify persistence
await page.reload();
await expect(page.getByText(agentName)).toBeVisible({ timeout: 10000 });
});
});
test.describe('when the weather tool is executed with a city', () => {
test('returns formatted weather data for that city', async () => {
// ARRANGE
await selectFixture(page, 'weather-success');
await page.goto('/tools/weather-tool');
// ACT: Execute tool with parameters
await page.getByTestId('param-city').fill('San Francisco');
await page.getByTestId('execute-tool-button').click();
// ASSERT: Verify OUTPUT content, not just that output appears
const output = page.getByTestId('tool-output');
await expect(output).toContainText('temperature');
await expect(output).toContainText('San Francisco');
// Verify structured data if applicable
const outputText = await output.textContent();
const outputData = JSON.parse(outputText || '{}');
expect(outputData).toHaveProperty('temperature');
expect(outputData).toHaveProperty('conditions');
});
});
test.describe('when a multi-step workflow is run', () => {
test('passes data between steps correctly', async () => {
// ARRANGE
await selectFixture(page, 'workflow-multi-step');
const sessionId = nanoid();
await page.goto(`/workflows/data-pipeline?session=${sessionId}`);
// ACT: Trigger workflow execution
await page.getByTestId('workflow-input').fill('test input data');
await page.getByTestId('run-workflow-button').click();
// ASSERT: Verify each step received correct input from previous step
// Wait for completion
await expect(page.getByTestId('workflow-status')).toHaveText('completed', { timeout: 30000 });
// Check step outputs show data transformation chain
const step1Output = await page.getByTestId('step-1-output').textContent();
const step2Output = await page.getByTestId('step-2-output').textContent();
// Verify step 2 received step 1's output as input
expect(step2Output).toContain(step1Output);
});
});
test.describe('when a multi-turn conversation is held', () => {
test('maintains conversation context across messages', async () => {
// ARRANGE
await selectFixture(page, 'contextual-chat');
const chatId = nanoid();
await page.goto(`/agents/assistant/chat/${chatId}`);
// ACT: Multi-turn conversation
await page.getByTestId('chat-input').fill('My name is Alice');
await page.getByTestId('send-button').click();
await expect(page.getByTestId('assistant-message').last()).toBeVisible({ timeout: 20000 });
await page.getByTestId('chat-input').fill('What is my name?');
await page.getByTestId('send-button').click();
// ASSERT: Verify context was maintained
const response = page.getByTestId('assistant-message').last();
await expect(response).toContainText('Alice', { timeout: 20000 });
});
});
test.describe('when the API fails during tool execution', () => {
test('shows an actionable error and allows a successful retry', async () => {
// ARRANGE: Set up failure fixture
await selectFixture(page, 'api-failure');
await page.goto('/tools/flaky-tool');
// ACT: Trigger the error
await page.getByTestId('execute-tool-button').click();
// ASSERT: Error is shown with recovery option
await expect(page.getByTestId('error-message')).toContainText('failed');
await expect(page.getByTestId('retry-button')).toBeVisible();
// Switch to success fixture and retry
await selectFixture(page, 'api-success');
await page.getByTestId('retry-button').click();
// Verify recovery worked
await expect(page.getByTestId('tool-output')).toBeVisible({ timeout: 10000 });
await expect(page.getByTestId('error-message')).not.toBeVisible();
});
});
When a test file already exists:
BEFORE (UI-focused):
test('dropdown opens when clicked', async () => {
await page.getByTestId('model-dropdown').click();
await expect(page.getByRole('listbox')).toBeVisible();
});
AFTER (Behavior-focused + BDD nesting):
test.describe('when a model is selected from the dropdown', () => {
test('updates and persists the agent configuration', async () => {
// Open dropdown and select model
await page.getByTestId('model-dropdown').click();
await page.getByRole('option', { name: 'GPT-4' }).click();
// Verify the selection persists and affects behavior
await page.reload();
await expect(page.getByTestId('model-dropdown')).toHaveText('GPT-4');
// Optionally: verify the model is used in actual requests
// (via request interception or checking response metadata)
});
});
Fixtures should represent realistic scenarios, not just mock data:
<feature>-<scenario>.fixture.ts
Examples:
- agent-with-tools.fixture.ts
- chat-multi-turn-context.fixture.ts
- workflow-parallel-execution.fixture.ts
- tool-validation-error.fixture.ts
- mcp-server-timeout.fixture.ts
Each fixture must define:
// fixtures/agent-provider-switch.fixture.ts
export const agentProviderSwitch = {
name: 'agent-provider-switch',
description: 'Tests that switching LLM providers changes agent behavior',
// Mock responses for different providers
responses: {
openai: { content: 'Response from OpenAI', model: 'gpt-4' },
anthropic: { content: 'Response from Anthropic', model: 'claude-3' },
},
expectedBehavior: {
// When provider is switched, subsequent messages use new provider
providerSwitchAffectsNextMessage: true,
// Provider selection persists across page reload
providerPersistsOnReload: true,
},
};
cd packages/playground && pnpm test:e2e
Before considering tests complete, verify:
test.describe names the unit under testtest is nested in a test.describe('when …') precondition block (no flat top-level test())test asserts exactly ONE observable outcomepage.reload() where applicable| Step | Command/Action |
|---|---|
| Build | pnpm build:cli |
| Start | cd packages/playground/e2e/kitchen-sink && pnpm dev |
| App URL | http://localhost:4111 |
| Routes | @packages/playground/src/App.tsx |
| Run tests | cd packages/playground && pnpm test:e2e |
| Test dir | packages/playground/e2e/tests/ |
| Fixtures | packages/playground/e2e/kitchen-sink/fixtures/ |
| ❌ Don't | ✅ Do Instead |
|---|---|
| Test that modal opens | Test that modal action completes and persists |
| Test that button is clickable | Test that clicking button produces expected result |
| Test loading spinner appears | Test that loaded data is correct |
| Test form validation message shows | Test that invalid form cannot submit AND valid form succeeds |
| Test dropdown has options | Test that selecting option changes system behavior |
| Test sidebar navigation works | Test that navigated page has correct data/functionality |
| Assert element is visible | Assert element contains expected data/state |
Top-level flat test() with no precondition describe | Nest every test in a test.describe('when …') block |
One test() asserting several unrelated outcomes | One test() per observable outcome |
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).