Implement saga patterns for distributed transactions and cross-aggregate workflows. Use when coordinating multi-step business processes, handling compensating transactions, or managing long-running workflows.
Saga definition with ordered steps, action commands, and compensation commands
Orchestrator or choreography implementation for your chosen pattern
Compensation logic for each participant service (idempotent, always-succeeds)
Step timeout configuration with per-step deadlines
Monitoring setup: state machine metrics, stuck saga detection, DLQ recovery
When to Use This Skill
Coordinating multi-service transactions without distributed locks
Implementing compensating transactions for partial failures
Managing long-running business workflows (minutes to hours)
Handling failures in distributed systems where atomicity is required
Building order fulfillment, approval, or booking processes
Replacing fragile two-phase commit with async compensation
Detailed section: Core Concepts
Moved to references/details.md.
Detailed section: Templates
Moved to references/details.md.
Best Practices
Do's
Make every step idempotent — Commands may be replayed on broker reconnect
Design compensations carefully — They are the most critical code path
Use correlation IDs — The saga_id must flow through every event and log
Implement per-step timeouts — Never wait indefinitely for a participant reply
Log state transitions — saga_id, step_name, old_state → new_state on every change
Test compensation paths explicitly — Inject failures at each step index in integration tests
Don'ts
Don't assume instant completion — Sagas are async and may take minutes
Don't skip compensation testing — The rollback path is the hardest to get right
Don't couple services directly — Use async messaging, never synchronous calls inside a saga step
Don't ignore partial failures — A step that partially executed still needs compensation
Don't use a global timeout — Each step has different latency characteristics
Troubleshooting
Saga stuck in COMPENSATING state
A saga enters compensation but never reaches FAILED. This means a compensation handler is throwing an unhandled exception and never publishing SagaCompensationCompleted. Add dead-letter queue (DLQ) handling to compensation consumers and ensure every compensation action publishes a result event even when the underlying operation was already rolled back.
If your orchestrator service restarts mid-saga, it may replay events and re-execute already-completed steps. Guard every step action with an idempotency key — see Template 3 above.
Choreography saga losing events
In a choreography-based saga, a downstream service may miss an event if it was offline when published. Use a durable message broker (Kafka with replication, RabbitMQ with persistence) and store the current saga state in a dedicated saga_log table so you can replay from the last known good step.
Timeout firing before a slow-but-valid step completes
A step like create_shipment might take up to 15 minutes during peak load but your global timeout is 5 minutes, causing spurious compensation. Make step timeouts configurable per step type — see references/advanced-patterns.md for the TimeoutSagaOrchestrator implementation and the STEP_TIMEOUTS dict pattern.
Compensation order not matching execution order
When two steps both complete before a failure is detected, compensation must run in strict reverse order or you leave data in an inconsistent state. Verify that _compensate() iterates from current_step - 1 down to 0, and add an integration test that deliberately fails at each step index to confirm correct rollback order.
Advanced Patterns
The references/ directory contains production-grade implementations not needed for most sagas:
references/advanced-patterns.md — Full SagaOrchestrator abstract base class, TimeoutSagaOrchestrator with per-step deadlines, detailed bank transfer compensating transaction chain, Prometheus instrumentation, stuck saga PromQL alerts, and DLQ recovery worker.
Related Skills
cqrs-implementation — Pair sagas with CQRS for read-model updates after each step completes
event-store-design — Store saga events in an event store for full audit trail and replay capability
workflow-orchestration-patterns — Higher-level workflow engines (Temporal, Conductor) that build on saga concepts