observability-guidelines

mindrally

•

6155 months ago

Source Raw

Observability guidelines for distributed systems using OpenTelemetry, tracing, metrics, and structured logging

Quick Install

npxskills add mindrally/skills--skill observability-guidelines

Instructions

Observability Guidelines

Apply these observability principles to ensure comprehensive visibility into distributed systems and microservices.

Core Observability Principles

Guide the development of idiomatic, maintainable, and high-performance code with built-in observability
Enforce modular design and separation of concerns through Clean Architecture
Promote test-driven development and robust observability from the start

OpenTelemetry Integration

Use OpenTelemetry for distributed tracing, metrics, and structured logging
Start and propagate tracing spans across all service boundaries
Use otel.Tracer for creating spans and otel.Meter for collecting metrics
Export data to OpenTelemetry Collector, Jaeger, or Prometheus
Configure appropriate sampling rates for production environments

Distributed Tracing

Trace all incoming requests and propagate context through internal calls
Use middleware to instrument HTTP and gRPC endpoints automatically
Include trace context in all downstream service calls
Create child spans for significant operations within a service
Add relevant attributes to spans for debugging and analysis

Metrics Collection

Monitor these key metrics across all services:

Request latency: Track p50, p90, p95, and p99 percentiles
Throughput: Measure requests per second by endpoint
Error rate: Track 4xx and 5xx responses separately
Resource usage: Monitor CPU, memory, disk, and network utilization
Custom business metrics: Track domain-specific KPIs

Structured Logging

Include unique request IDs and trace context in all logs for correlation
Use structured logging formats (JSON) for machine parseability
Include relevant context: timestamp, service name, trace ID, span ID
Log at appropriate levels: DEBUG, INFO, WARN, ERROR
Avoid logging sensitive information (PII, credentials)

Architecture Patterns

Apply Clean Architecture with handlers, services, repositories, and domain models
Use domain-driven design principles for clear boundaries
Prioritize interface-driven development with explicit dependency injection
Prefer composition over inheritance; favor small, purpose-specific interfaces

Correlation and Context

Propagate context through the entire request lifecycle
Use correlation IDs for request tracking across services
Include service version and deployment information in telemetry
Tag traces with relevant business context for filtering
Enable trace-to-log and log-to-trace correlation

Alerting and Dashboards

Create dashboards for service health and business metrics
Set up alerts based on SLOs and error budgets
Use anomaly detection for proactive issue identification
Document runbooks for common alert scenarios
Review and tune alerts regularly to reduce noise

Instrumentation Best Practices

Instrument at service boundaries (entry/exit points)
Add custom spans for database operations and external calls
Include relevant attributes (user ID, request type, etc.)
Avoid over-instrumentation that creates noise
Use semantic conventions for consistent attribute naming

Production Considerations

Configure appropriate sampling rates to balance visibility and cost
Use head-based sampling for consistent trace capture
Implement tail-based sampling for capturing errors
Set retention policies based on debugging needs
Monitor observability infrastructure health

Loading…

Related Skills

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,360

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,309

1password

Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.

365,915

himalaya

CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).

365,885