Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
npxskills add dotnet/maui--skill verify-tests-fail-without-fixLoading…
Verifies tests actually catch the issue. Supports all test types (UI tests, unit tests, XAML tests, device tests) and two workflow modes.
| Test Type | Auto-Detected From | Runner |
|---|---|---|
| UITest | TestCases.Shared.Tests/, TestCases.HostApp/ | BuildAndRunHostApp.ps1 |
| DeviceTest | DeviceTests/ | Run-DeviceTests.ps1 |
| UnitTest | *.UnitTests/, |
Graphics.Tests/dotnet test |
| XamlUnitTest | Xaml.UnitTests/ | dotnet test |
Test type is auto-detected from changed files. Override with -TestType if needed.
-Platform is required for UI and Device tests. It selects which platform to verify the fix on. Unit and XAML tests do not require -Platform.
🛑 This skill ONLY verifies that existing tests reproduce a bug. Do NOT activate for:
Requires: a platform and either test files in the PR or an explicit TestFilter.
In this skill, test outcomes mean the OPPOSITE of normal:
| Test Result (without fix) | Verification Result | Why |
|---|---|---|
| Tests FAIL | ✅ GOOD | Tests detect the bug |
| Tests PASS | ❌ BAD | Tests miss the bug |
NEVER say "verification passed" when tests PASS without the fix.
-RequireFullVerification)pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 `
-Platform <platform> `
-TestFilter "<filter>" `
[-RequireFullVerification] # Only if fix files exist
⚠️ Remember: test outcomes are INVERTED from normal!
VERIFICATION PASSED → Tests catch the bug ✅VERIFICATION FAILED → Tests don't catch the bug ❌Use when creating tests before writing a fix:
# Auto-detect test type and filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android
# Explicit test type + filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -TestType UnitTest -TestFilter "Maui12345"
Use when validating both tests and fix:
# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification
# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification
Note: -RequireFullVerification ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.
Verify Failure Only Mode:
Full Verification Mode:
The script auto-detects which mode to use based on whether fix files are present.
Verify Failure Only Mode:
╔═══════════════════════════════════════════════════════════╗
║ VERIFICATION PASSED ✅ ║
╠═══════════════════════════════════════════════════════════╣
║ Tests FAILED as expected! ║
║ This proves the tests correctly reproduce the bug. ║
╚═══════════════════════════════════════════════════════════╝
Full Verification Mode:
╔═══════════════════════════════════════════════════════════╗
║ VERIFICATION PASSED ✅ ║
╠═══════════════════════════════════════════════════════════╣
║ - FAIL without fix (as expected) ║
║ - PASS with fix (as expected) ║
╚═══════════════════════════════════════════════════════════╝
Verify Failure Only Mode (no fix files):
Full Verification Mode (fix files detected):
CustomAgentLogsTmp/TestValidation/verification-report.md - Full detailed reportCustomAgentLogsTmp/PRState/verification-report.md - Validate section for agentNote: PR label management (s/ai-reproduction-confirmed / s/ai-reproduction-failed) is handled by Review-PR.ps1, not by this script.
The skill generates output files under CustomAgentLogsTmp/PRState/<PRNumber>/PRAgent/gate/verify-tests-fail/:
| File | Description |
|---|---|
verification-report.md | Comprehensive markdown report with test results and full logs |
verification-log.txt | Text log of the verification process |
test-without-fix.log | Full test output from run without fix |
test-with-fix.log | Full test output from run with fix |
Plus test logs in CustomAgentLogsTmp/:
UITests/ - UI test device logs and outputDeviceTests/ - Device test outputUnitTests/ - Unit test outputExample structure:
CustomAgentLogsTmp/
├── UITests/ # UI test logs
│ ├── android-device.log
│ └── test-output.log
├── DeviceTests/ # Device test logs
│ └── test-output.log
├── UnitTests/ # Unit/XAML test logs
│ └── test-output.log
└── PRState/
└── 27847/
└── PRAgent/
└── gate/
└── verify-tests-fail/
├── verification-report.md # Full detailed report
├── verification-log.txt
├── test-without-fix.log
└── test-with-fix.log
PR Number Detection:
pr-27847)gh pr view command-PRNumber parameter| Problem | Cause | Solution |
|---|---|---|
| No fix files detected | Base branch detection failed or no non-test files changed | Use -FixFiles or -BaseBranch explicitly |
| Tests pass without fix | Tests don't detect the bug | Review test assertions, update test |
| Tests fail with fix | Fix doesn't work or test is wrong | Review fix implementation |
| App crashes | Duplicate issue numbers, XAML error | Check device logs |
| Element not found | Wrong AutomationId, app crashed | Verify IDs match |
# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification
# Explicit test type (auto-detected if omitted)
-TestType UnitTest # or XamlUnitTest, DeviceTest, UITest
# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"
# Explicit fix files
-FixFiles @("src/Core/src/File.cs")
# Explicit base branch
-BaseBranch "main"
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).