Skip to main content
openevidence-observability Set up comprehensive observability for OpenEvidence integrations with metrics, traces, and alerts.
Use when implementing monitoring for clinical AI operations, setting up dashboards,
or configuring alerting for healthcare application health.
Trigger with phrases like "openevidence monitoring", "openevidence metrics",
"openevidence observability", "monitor openevidence", "openevidence alerts".
npx skills add jeremylongshore/claude-code-plugins-plus-skills --skill openevidence-observability ai automation claude-code devops mcp ai-agents
OpenEvidence Observability
Overview
OpenEvidence delivers clinical evidence queries where response accuracy and freshness have direct patient safety implications. Monitor query response times to ensure clinicians get timely answers, track evidence freshness to catch stale citations, and audit every query for compliance. Observability must also verify citation accuracy and maintain complete audit logs for regulatory requirements (HIPAA, clinical decision support standards).
Key Metrics
Metric Type Target Alert Threshold Query response time p95 Histogram < 3s > 8s Evidence freshness Gauge < 7 days median > 30 days Citation accuracy rate Gauge > 95% < 90% API error rate Gauge < 0.5% > 2% Audit log completeness Gauge 100% < 99.9% Daily query volume
Instrumentation async function trackClinicalQuery(queryType: string, fn: () => Promise<any>) {
const start = Date.now();
const traceId = crypto.randomUUID();
try {
const result = await fn();
metrics.histogram('openevidence.query.latency', Date.now() - start, { queryType });
metrics.increment('openevidence.query.total', { queryType });
auditLog.record({ traceId, queryType, status: 'ok', latency: Date.now() - start });
return result;
} catch (err) {
metrics.increment('openevidence.query.errors', { queryType, error: err.code });
auditLog.record({ traceId, queryType, status: 'error', error: err.message });
throw err;
}
}
Health Check Dashboard async function openEvidenceHealth(): Promise<Record<string, string>> {
const latencyP95 = await metrics.query('openevidence.query.latency', 'p95', '5m');
const errorRate = await metrics.query('openevidence.query.error_rate', 'avg', '5m');
const freshness = await openEvAdmin.getMedianEvidenceAge();
return {
query_latency: latencyP95 < 3000 ? 'healthy' : 'slow',
error_rate: errorRate < 0.005 ? 'healthy' : 'degraded',
evidence_freshness: freshness < 7 ? 'healthy' : 'stale',
};
}
Alerting Rules const alerts = [
{ metric: 'openevidence.query.latency_p95', condition: '> 8s', window: '10m', severity: 'warning' },
{ metric: 'openevidence.query.error_rate', condition: '> 0.02', window: '5m', severity: 'critical' },
{ metric: 'openevidence.evidence.median_age_days', condition: '> 30', window: '1d', severity: 'warning' },
{ metric: 'openevidence.audit.completeness', condition: '< 0.999', window: '1h', severity: 'critical' },
];
Structured Logging function logClinicalEvent(event: string, data: Record<string, any>) {
console.log(JSON.stringify({
service: 'openevidence', event,
query_type: data.queryType, duration_ms: data.latency,
citation_count: data.citations, evidence_age_days: data.evidenceAge,
// HIPAA: never log patient identifiers or query text
trace_id: data.traceId, audit_seq: data.auditSeq,
timestamp: new Date().toISOString(),
}));
}
Error Handling Signal Meaning Action Query timeout > 8s Evidence index overloaded Check index health, scale read replicas Citation accuracy drop Stale or retracted sources Trigger evidence refresh pipeline Audit log gap Logging pipeline failure Critical — investigate immediately for compliance 429 rate limit Quota approaching limit Throttle non-critical queries, request increase Evidence age > 30 days Refresh pipeline stalled Check ingestion jobs, verify source feeds
Resources
Next Steps See openevidence-incident-runbook.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
