Validate AI/ML models and datasets for bias, fairness, and ethical concerns. Use when auditing AI systems for ethical compliance, fairness assessment, or bias detection. Trigger with phrases like "evaluate model fairness", "check for bias", or "validate AI ethics".
npxskills add jeremylongshore/claude-code-plugins-plus-skills--skill validating-ai-ethics-and-fairnessLoading…
Validate AI/ML models and datasets for bias, fairness, and ethical compliance using quantitative fairness metrics and structured audit workflows.
pip install fairlearn)pip install aif360) for comprehensive bias analysisExponentiatedGradient from Fairlearn)| Error | Cause | Solution |
|---|---|---|
| Insufficient group sample size | Fewer than 30 observations in a demographic group | Aggregate related subgroups; use bootstrap confidence intervals; flag metric as unreliable |
| Missing sensitive attributes | Protected attribute columns absent from dataset | Apply proxy detection via correlated features; request attribute access under data governance approval |
| Conflicting fairness criteria | Demographic parity and equalized odds contradict | Document the impossibility theorem trade-off; prioritize the metric most aligned with the deployment context |
| Data quality failures | Inconsistent encoding or null values in attribute columns | Standardize categorical encodings; impute or exclude nulls; validate with schema checks before analysis |
| Model output format mismatch | Predictions not in expected probability or binary format | Convert logits to probabilities via sigmoid; binarize at the decision threshold before metric computation |
Scenario 1: Hiring Model Audit -- Validate a resume-screening classifier for gender and age bias. Compute demographic parity across male/female groups and age buckets (18-30, 31-50, 51+). Apply the four-fifths rule. Finding: female selection rate at 0.72 of male rate (critical severity). Recommend reweighting training samples and adjusting the decision threshold.
Scenario 2: Credit Scoring Fairness -- Assess a credit approval model for racial disparate impact. Calculate equalized odds (TPR and FPR) across racial groups. Finding: FPR for Group A is 2.1x Group B (high severity). Recommend in-processing constraint using ExponentiatedGradient with FalsePositiveRateParity.
Scenario 3: Healthcare Risk Prediction -- Evaluate a patient risk model for age and socioeconomic bias. Compute calibration curves per group. Finding: model overestimates risk for low-income patients by 15%. Recommend recalibration using Platt scaling per subgroup with post-deployment monitoring for fairness drift.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).