When working on a category of CTS tests, follow this systematic process to identify issues, prioritize fixes, and document findings.
List all the tests matching a selector:
cargo xtask cts -- --list 'webgpu:api,validation,*' 2>&1 | wc -l
If there is a reasonable number of tests matching the selector (less than a couple hundred or so, but this isn't a hard cutoff), then you can proceed with triage. Otherwise, make a list of more detailed wildcards that match fewer tests, verify that each wildcard matches a reasonable number of tests, and triage each wildcard separately.
Run the full test suite for the category to understand the scope:
cargo xtask cts 'webgpu:api,validation,category:*' 2>&1 | grep -E "(Summary|Passed|Failed)" | tail -5
This gives you the pass rate and number of failures. Document this as your baseline.
Review the output from the running the CTS (with or without --list) to
identify any subcategories that may exist within the suite being analyzed.
Subcategories typically have an additional :- or ,-delimited word in the
test name. Running tests by subcategory may be more manageable than running
with the entire suite at once or running individual tests.
Test each subcategory individually to identify which ones are passing vs failing:
cargo xtask cts 'webgpu:api,validation,category:subcategory:*' 2>&1 | grep -E "(Passed|Failed|Skipped)" | tail -3
Track the pass rate for each subcategory. This helps you identify:
For failing subcategories, look at what tests are failing:
cargo xtask cts 'webgpu:api,validation,category:subcategory:*' 2>&1 | grep "\[fail\]" | head -20
Look for patterns:
Pick a representative failing test and run it individually to see the error:
cargo xtask cts 'webgpu:api,validation,category:subcategory:specific_test' 2>&1 | tail -30
Look for:
To understand what the test expects, read the TypeScript source:
grep -A 40 "test_name" cts/src/webgpu/api/validation/path/file.spec.ts
The test source shows:
Group failures into categories:
High Priority - Validation Gaps:
Medium Priority - Spec Compliance:
Low Priority - Minor Gaps:
Known Issues - Skip:
For validation gaps, find where validation should happen:
Search for existing validation:
grep -n "relevant_keyword" wgpu-core/src/device/resource.rs
Look for render/compute pipeline creation:
wgpu-core/src/device/resource.rs around create_render_pipelineCheck for helper functions:
grep "fn is_" wgpu-types/src/texture/format.rs
Find error enums:
grep "pub enum.*Error" wgpu-core/src/pipeline.rs
When implementing fixes:
wgpu-core/src/pipeline.rs)wgpu-types if checking properties)wgpu-core/src/device/resource.rs)Create or update a triage document (e.g., category_triage.md).
Do not write information about changes you have made to the triage document. Only capture the state of the tests and any investigation into open issues.
# Category CTS Tests - Triage Report
**Overall Status:** XP/YF/ZS (%/%/%)
## Passing Sub-suites ✅
[List sub-suites that have no failures (all pass or skip)]
## Remaining Issues ⚠️
[List sub-suites that have failures and if it can be stated concisely, a summary of the issue]
## Issue Detail
[List detail of any investigation into failures. Do not go into detail about passed suites, just list the failures.]
### 1. title, e.g. a distinguishing word from the test selector
**Test selector:** `webgpu:api,validation,render_pipeline,depth_stencil_state:format:*`
**What it tests:** [Description]
**Example failure:**
[a selector for a single failing test, e.g.:]
```
webgpu:api,validation,render_pipeline,depth_stencil_state:depthCompare_optional:isAsync=false;format="stencil8"
```
**Error:**
[error message from the failing tests, e.g.:]
```
Unexpected validation error occurred: Depth/stencil state is invalid:
Format Stencil8 does not have a depth aspect, but depth test/write is enabled
```
**Root cause:**
[Your analysis of the root cause. Do not speculate. Only include the results of specific investigation you have done.]
The validation is triggering incorrectly. When `depthCompare` is undefined/missing in JavaScript, it's getting translated to a default value that makes `is_depth_enabled()` return true, even for stencil-only formats.
**Fix needed:**
[Your proposed fix. Again, do not speculate. Only state the fix if it is obvious from the root cause analysis, or if you have done specific investigation into how to fix it.]
### 2. title
[repeat as needed for additional issues]
For fixed tests that are now passing, add them to cts_runner/test.lst:
Use wildcards to minimize lines:
webgpu:api,validation,category:subcategory:isAsync=false;*
Group related tests when possible:
webgpu:api,validation,category:* if all subcategories passMaintain alphabetical order roughly in the file
Before finishing:
# Format code
cargo fmt
# Check for errors
cargo clippy --tests
# Build to ensure no compilation errors
cargo build
# Run the tests you added to test.lst
cargo xtask cts 'webgpu:api,validation,category:subcategory:isAsync=false;*'
If the user asked to investigate a failure, and the cause of the failure is noted here with "do not attempt to fix", then stop and ask the user before attempting to fix.
Pattern: Format-specific failures
is_depth_stencil_format(), is_color_format() etc.Pattern: Aspect-related failures
hal::FormatAspects::from(format) to checkPattern: Optional field failures
Pattern: Atomics accepted incorrectly
atomicLoad, atomicStore, etc.Pattern: Error reporting for destroyed resources
wgpu often reports these errors later than WebGPU requires, causing the tests to fail.wgpu may report these errors earlier than it should, causing the test to fail with an unexpected validation error.state="destroyed" parameterwebgpu:api,validation,encoding,cmds,compute_pass:indirect_dispatch_buffer_state: with state="destroyed" subcases
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).