dotnet
Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
bunx add-skill dotnet/maui -s verify-tests-fail-without-fixLoadingβ¦
Verifies UI tests actually catch the issue. Supports two workflow modes:
Use when creating tests before writing a fix:
# Auto-detect test filter from changed test files
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android
# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356"
Use when validating both tests and fix:
# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification
# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification
ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.
-RequireFullVerificationVerify Failure Only Mode:
Full Verification Mode:
The script auto-detects which mode to use based on whether fix files are present.
Verify Failure Only Mode:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β VERIFICATION PASSED β
β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ£
β Tests FAILED as expected! β
β This proves the tests correctly reproduce the bug. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Full Verification Mode:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β VERIFICATION PASSED β
β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ£
β - FAIL without fix (as expected) β
β - PASS with fix (as expected) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Verify Failure Only Mode (no fix files):
Full Verification Mode (fix files detected):
TestCases.Shared.Tests/*.csCustomAgentLogsTmp/TestValidation/verification-report.md - Full detailed reportCustomAgentLogsTmp/PRState/verification-report.md - Gate section for PR agentThe skill automatically manages two labels on the PR to indicate verification status:
| Label | Color | When Applied |
|---|---|---|
s/ai-reproduction-confirmed | π’ Green (#2E7D32) | Tests correctly FAIL without fix (AI verified tests catch the bug) |
s/ai-reproduction-failed | π Orange (#E65100) | Tests PASS without fix (AI verified tests don't catch the bug) |
Behavior:
s/ai-reproduction-confirmed and removes s/ai-reproduction-failed if presents/ai-reproduction-failed and removes s/ai-reproduction-confirmed if presentThe skill generates output files under CustomAgentLogsTmp/PRState/<PRNumber>/verify-tests-fail/:
| File | Description |
|---|---|
verification-report.md | Comprehensive markdown report with test results and full logs |
verification-log.txt | Text log of the verification process |
test-without-fix.log | Full test output from run without fix |
test-with-fix.log | Full test output from run with fix |
Plus UI test logs in CustomAgentLogsTmp/UITests/:
android-device.log or ios-device.log - Device logstest-output.log - NUnit test outputExample structure:
CustomAgentLogsTmp/
βββ UITests/ # Shared UI test logs
β βββ android-device.log
β βββ test-output.log
βββ PRState/
βββ 27847/
βββ verify-tests-fail/
βββ verification-report.md # Full detailed report
βββ verification-log.txt
βββ test-without-fix.log
βββ test-with-fix.log
PR Number Detection:
pr-27847)gh pr view command-PRNumber parameter| Problem | Cause | Solution |
|---|---|---|
| No fix files detected | Base branch detection failed or no non-test files changed | Use -FixFiles or -BaseBranch explicitly |
| Tests pass without fix | Tests don't detect the bug | Review test assertions, update test |
| Tests fail with fix | Fix doesn't work or test is wrong | Review fix implementation |
| App crashes | Duplicate issue numbers, XAML error | Check device logs |
| Element not found | Wrong AutomationId, app crashed | Verify IDs match |
# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification
# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"
# Explicit fix files
-FixFiles @("src/Core/src/File.cs")
# Explicit base branch
-BaseBranch "main"
Use when you want to validate changes before committing, or when you need to check all React contribution requirements.
Use when you have lint errors, formatting issues, or before committing code to ensure it passes CI.
Use when you need to run Flow type checking, or when seeing Flow type errors in React code.
Use when feature flag tests fail, flags need updating, understanding @gate pragmas, debugging channel-specific test failures, or adding new flags to React.