Skip to main content
openevidence-prod-checklist Production readiness checklist for OpenEvidence clinical AI deployments.
Use when preparing for production launch, conducting pre-deployment reviews,
or auditing OpenEvidence integration for compliance.
Trigger with phrases like "openevidence production", "openevidence go-live",
"openevidence checklist", "deploy openevidence", "openevidence readiness".
npx skills add jeremylongshore/claude-code-plugins-plus-skills --skill openevidence-prod-checklist ai automation claude-code devops mcp ai-agents
OpenEvidence Production Checklist
Overview
OpenEvidence provides clinical decision support backed by peer-reviewed medical literature. A production integration handles Protected Health Information (PHI) subject to HIPAA, serves evidence-based answers where accuracy directly impacts patient outcomes, and must maintain complete audit trails for regulatory review. Misconfigurations can expose PHI in logs, serve stale clinical guidance, or fail compliance audits that shut down your integration entirely. This checklist enforces HIPAA-grade security, citation verification, and the SLA discipline required for healthcare-adjacent systems.
Prerequisites
Production OpenEvidence API credentials (not trial/sandbox keys)
Secrets manager configured (Vault, AWS Secrets Manager, or GCP Secret Manager)
HIPAA-compliant monitoring stack (no PHI in log aggregators without BAA)
Business Associate Agreement (BAA) executed with OpenEvidence
Compliance officer sign-off on data flow architecture
Authentication & Secrets
API Integration
Error Handling & Resilience
Monitoring & Alerting
Security
Validation Script async function validateOpenEvidenceProduction(apiKey: string): Promise<void> {
const base = process.env.OPENEVIDENCE_API_URL ?? 'https://api.openevidence.com/v1';
const headers = { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' };
// 1. Connectivity check
const ping = await fetch(`${base}/health`, { headers, signal: AbortSignal.timeout(5000) });
console.assert(ping.ok, `API unreachable: ${ping.status}`);
// 2. Auth validation
const auth = await fetch(`${base}/me`, { headers });
console.assert(auth.status !== 401, 'Invalid API key');
console.assert(auth.status !== 403, 'Insufficient permissions — check scope');
// 3. Clinical query round-trip (de-identified test query)
const query = await fetch(`${base}/query`, {
method: 'POST',
headers,
body: JSON.stringify({ question: 'What is the standard treatment for hypertension?' }),
signal: AbortSignal.timeout(15000),
});
console.assert(query.ok, `Clinical query failed: ${query.status}`);
const result = await query.json();
console.assert(result.citations?.length > 0, 'No citations returned — evidence pipeline may be down');
// 4. Response time SLA
const start = Date.now();
await fetch(`${base}/query`, {
method: 'POST',
headers,
body: JSON.stringify({ question: 'Recommended dosage for metformin in type 2 diabetes?' }),
signal: AbortSignal.timeout(15000),
});
const elapsed = Date.now() - start;
console.assert(elapsed < 3000, `Response time ${elapsed}ms exceeds 3s SLA`);
// 5. Audit log endpoint accessible
const audit = await fetch(`${base}/audit-log?limit=1`, { headers });
console.assert(audit.ok, `Audit log endpoint failed: ${audit.status}`);
console.log('All OpenEvidence production checks passed');
}
Risk Matrix Check Risk if Skipped Priority PHI excluded from API payloads HIPAA violation, regulatory penalty, BAA breach Critical PHI excluded from logs Data breach via log aggregator, OCR enforcement action Critical Audit log completeness Failed compliance audit, integration shutdown Critical Citation URL validation Clinicians follow broken links, lose trust in evidence High Confidence score monitoring Low-quality answers served without clinician awareness High
Resources
Next Steps See openevidence-security-basics.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
