Skip to main content
workflow-orchestration-patterns Design durable workflows with Temporal for distributed systems. Covers workflow vs activity separation, saga patterns, state management, and determinism constraints. Use when building long-running processes, distributed transactions, or microservice orchestration.
npx skills add sickn33/antigravity-awesome-skills --skill workflow-orchestration-patterns agentic-skills ai-agents antigravity claude-code mcp ai-workflows
Workflow Orchestration Patterns
Master workflow orchestration architecture with Temporal, covering fundamental design decisions, resilience patterns, and best practices for building reliable distributed systems.
Use this skill when
Working on workflow orchestration patterns tasks or workflows
Needing guidance, best practices, or checklists for workflow orchestration patterns
Do not use this skill when
The task is unrelated to workflow orchestration patterns
You need a different domain or tool outside this scope
Instructions
Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open resources/implementation-playbook.md.
When to Use Workflow Orchestration
Ideal Use Cases (Source: docs.temporal.io)
Multi-step processes spanning machines/services/databasesDistributed transactions requiring all-or-nothing semanticsLong-running workflows (hours to years) with automatic state persistenceFailure recovery that must resume from last successful stepBusiness processes : bookings, orders, campaigns, approvalsEntity lifecycle management : inventory tracking, account management, cart workflows
Infrastructure automation
Human-in-the-loop systems requiring timeouts and escalations
When NOT to Use
Simple CRUD operations (use direct API calls)
Pure data processing pipelines (use Airflow, batch processing)
Stateless request/response (use standard APIs)
Real-time streaming (use Kafka, event processors)
Critical Design Decision: Workflows vs Activities The Fundamental Rule (Source: temporal.io/blog/workflow-engine-principles):
Workflows = Orchestration logic and decision-makingActivities = External interactions (APIs, databases, network calls)
Workflows (Orchestration)
Contain business logic and coordination
MUST be deterministic (same inputs → same outputs)Cannot perform direct external callsState automatically preserved across failures
Can run for years despite infrastructure failures
Decide which steps to execute
Handle compensation logic
Manage timeouts and retries
Coordinate child workflows
Activities (External Interactions)
Handle all external system interactions
Can be non-deterministic (API calls, DB writes)
Include built-in timeouts and retry logic
Must be idempotent (calling N times = calling once)Short-lived (seconds to minutes typically)
Call payment gateway API
Write to database
Send emails or notifications
Query external services
Design Decision Framework Does it touch external systems? → Activity
Is it orchestration/decision logic? → Workflow
Core Workflow Patterns
1. Saga Pattern with Compensation Purpose : Implement distributed transactions with rollback capability
Pattern (Source: temporal.io/blog/compensating-actions-part-of-a-complete-breakfast-with-sagas):
For each step:
1. Register compensation BEFORE executing
2. Execute the step (via activity)
3. On failure, run all compensations in reverse order (LIFO)
Example: Payment Workflow
Reserve inventory (compensation: release inventory)
Charge payment (compensation: refund payment)
Fulfill order (compensation: cancel fulfillment)
Compensations must be idempotent
Register compensation BEFORE executing step
Run compensations in reverse order
Handle partial failures gracefully
2. Entity Workflows (Actor Model) Purpose : Long-lived workflow representing single entity instance
Pattern (Source: docs.temporal.io/evaluate/use-cases-design-patterns):
One workflow execution = one entity (cart, account, inventory item)
Workflow persists for entity lifetime
Receives signals for state changes
Supports queries for current state
Shopping cart (add items, checkout, expiration)
Bank account (deposits, withdrawals, balance checks)
Product inventory (stock updates, reservations)
Encapsulates entity behavior
Guarantees consistency per entity
Natural event sourcing
3. Fan-Out/Fan-In (Parallel Execution) Purpose : Execute multiple tasks in parallel, aggregate results
Spawn child workflows or parallel activities
Wait for all to complete
Aggregate results
Handle partial failures
Scaling Rule (Source: temporal.io/blog/workflow-engine-principles):
Don't scale individual workflows
For 1M tasks: spawn 1K child workflows × 1K tasks each
Keep each workflow bounded
4. Async Callback Pattern Purpose : Wait for external event or human approval
Workflow sends request and waits for signal
External system processes asynchronously
Sends signal to resume workflow
Workflow continues with response
Human approval workflows
Webhook callbacks
Long-running external processes
State Management and Determinism
Automatic State Preservation How Temporal Works (Source: docs.temporal.io/workflows):
Complete program state preserved automatically
Event History records every command and event
Seamless recovery from crashes
Applications restore pre-failure state
Determinism Constraints Workflows Execute as State Machines :
Replay behavior must be consistent
Same inputs → identical outputs every time
Prohibited in Workflows (Source: docs.temporal.io/workflows):
❌ Threading, locks, synchronization primitives
❌ Random number generation (random())
❌ Global state or static variables
❌ System time (datetime.now())
❌ Direct file I/O or network calls
❌ Non-deterministic libraries
✅ workflow.now() (deterministic time)
✅ workflow.random() (deterministic random)
✅ Pure functions and calculations
✅ Calling activities (non-deterministic operations)
Versioning Strategies Challenge : Changing workflow code while old executions still running
Versioning API : Use workflow.get_version() for safe changesNew Workflow Type : Create new workflow, route new executions to itBackward Compatibility : Ensure old events replay correctly
Resilience and Error Handling
Retry Policies Default Behavior : Temporal retries activities forever
Initial retry interval
Backoff coefficient (exponential backoff)
Maximum interval (cap retry delay)
Maximum attempts (eventually fail)
Invalid input (validation failures)
Business rule violations
Permanent failures (resource not found)
Idempotency Requirements Why Critical (Source: docs.temporal.io/activities):
Activities may execute multiple times
Network failures trigger retries
Duplicate execution must be safe
Implementation Strategies :
Idempotency keys (deduplication)
Check-then-act with unique constraints
Upsert operations instead of insert
Track processed request IDs
Activity Heartbeats Purpose : Detect stalled long-running activities
Activity sends periodic heartbeat
Includes progress information
Timeout if no heartbeat received
Enables progress-based retry
Best Practices
Workflow Design
Keep workflows focused - Single responsibility per workflowSmall workflows - Use child workflows for scalabilityClear boundaries - Workflow orchestrates, activities executeTest locally - Use time-skipping test environment
Activity Design
Idempotent operations - Safe to retryShort-lived - Seconds to minutes, not hoursTimeout configuration - Always set timeoutsHeartbeat for long tasks - Report progressError handling - Distinguish retryable vs non-retryable
Common Pitfalls
Using datetime.now() instead of workflow.now()
Threading or async operations in workflow code
Calling external APIs directly from workflow
Non-deterministic logic in workflows
Non-idempotent operations (can't handle retries)
Missing timeouts (activities run forever)
No error classification (retry validation errors)
Ignoring payload limits (2MB per argument)
Operational Considerations
Workflow execution duration
Activity failure rates
Retry attempts and backoff
Pending workflow counts
Horizontal scaling with workers
Task queue partitioning
Child workflow decomposition
Activity batching when appropriate
Additional Resources
Temporal Core Concepts: docs.temporal.io/workflows
Workflow Patterns: docs.temporal.io/evaluate/use-cases-design-patterns
Best Practices: docs.temporal.io/develop/best-practices
Saga Pattern: temporal.io/blog/saga-pattern-made-easy
Workflows = orchestration, Activities = external calls
Determinism is non-negotiable for workflows
Idempotency is critical for activities
State preservation is automatic
Design for failure and recovery
Limitations
Use this skill only when the task clearly matches the scope described above.
Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
