Temporal Workflow Orchestration in Go: Signals, Timers, and Saga Compensation

Distributed transactions are hard. When your checkout flow spans inventory reservation, payment charging, warehouse notification, and email delivery — any step can fail, the process can crash mid-way, and you need to undo completed steps on failure. Most teams reach for queues and flags in a database. Temporal is a better answer.

This post walks through a real Go service using Temporal to orchestrate a full checkout flow: inventory reservation, payment via webhook signal, delivery confirmation, and automatic compensation on failure.

What is Temporal?

Temporal is a durable workflow execution platform. You write ordinary Go code — functions, loops, conditionals — and Temporal makes it fault-tolerant and resumable:

• Workflow state survives process restarts (event sourcing under the hood)
• Activities automatically retry with backoff on failure
• Timers survive crashes — a 30-minute timer set before a crash fires correctly after restart
• Full execution history visible in the Temporal UI

The mental model: your workflow function is never truly interrupted. Temporal replays history to restore state after any failure.

Orchestration vs Choreography

Before the code — a key architectural choice.

Choreography (event-driven pub/sub):

OrderSvc  → publish "OrderCreated"
PaymentSvc → listen → charge → publish "PaymentDone"
WarehouseSvc → listen → pack → publish "Packed"
NotifSvc  → listen → send email

Each service reacts to events independently. Needs a message broker. Distributed logic is hard to trace and reason about.

Orchestration (Temporal):

// CheckoutWorkflow is the single source of truth for the entire flow
workflow.ExecuteActivity(ctx, acts.CheckoutReserveInventory, input)
workflow.ExecuteActivity(ctx, acts.CheckoutChargePayment, payment)
workflow.ExecuteActivity(ctx, acts.CheckoutNotifyWarehouse, input.OrderID)
workflow.ExecuteActivity(ctx, acts.CheckoutSendConfirmEmail, input.OrderID)

One workflow function drives the entire flow. The orchestrator knows every step, handles retries, and can compensate on failure. Full history in Temporal UI.

Temporal shines for long-running business transactions where you need durability, compensation, and observability.

Project Structure

Using DDD (Domain-Driven Design) with one module per domain:

modules/checkout/
├── application/
│   ├── dto/           # HTTP request/response structs
│   └── services/      # CheckoutService — wires repo + Temporal client
├── domain/
│   ├── entities/      # Checkout, PaymentInfo, DeliveryInfo
│   └── interfaces/    # CheckoutService, CheckoutRepository, IdempotencyStore
└── infrastructure/
    ├── fake/          # FakeInventoryService, FakePaymentGateway, etc.
    ├── repositories/  # in-memory checkout store
    └── temporal/      # CheckoutWorkflow + Activities

Each layer depends only inward. The workflow lives in infrastructure/temporal/ — infrastructure detail, not domain logic.

The Checkout Workflow

This is the full flow: reserve inventory → wait for payment (webhook signal or 30-min timeout) → charge → confirm → wait for delivery (30-day timeout) → send review request.

const (
    TaskQueue               = "checkout-task-queue"
    PaymentReceivedSignal   = "payment-received"
    DeliverySignal          = "delivery-confirmed"
    PaymentTimeoutDuration  = 30 * time.Minute
    DeliveryTimeoutDuration = 30 * 24 * time.Hour
)

func CheckoutWorkflow(ctx workflow.Context, input CheckoutInput) (*CheckoutResult, error) {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 3,
            InitialInterval: time.Second,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var actRef Activities // nil stub — Temporal resolves by method name via reflection

    // 1. Reserve inventory immediately
    if err := workflow.ExecuteActivity(ctx, actRef.CheckoutReserveInventory, input).Get(ctx, nil); err != nil {
        return nil, err
    }

    // 2. Race: payment signal vs 30-min timeout
    var paymentInfo PaymentSignalPayload
    timerFired := false

    sel := workflow.NewSelector(ctx)
    sel.AddReceive(workflow.GetSignalChannel(ctx, PaymentReceivedSignal),
        func(ch workflow.ReceiveChannel, _ bool) {
            ch.Receive(ctx, &paymentInfo)
        })
    sel.AddFuture(workflow.NewTimer(ctx, PaymentTimeoutDuration),
        func(_ workflow.Future) { timerFired = true })
    sel.Select(ctx) // blocks until one fires

    // 3. Timeout path — compensate + notify
    if timerFired {
        workflow.ExecuteActivity(ctx, actRef.CheckoutReleaseInventory, input.OrderID, input.UserID).Get(ctx, nil)
        workflow.ExecuteActivity(ctx, actRef.CheckoutSendTimeoutEmail, input.OrderID, input.UserID).Get(ctx, nil)
        return &CheckoutResult{OrderID: input.OrderID, Status: "timeout"}, nil
    }

    // 4. Charge payment — compensate on failure (saga pattern)
    if err := workflow.ExecuteActivity(ctx, actRef.CheckoutChargePayment, paymentInfo).Get(ctx, nil); err != nil {
        workflow.ExecuteActivity(ctx, actRef.CheckoutReleaseInventory, input.OrderID, input.UserID).Get(ctx, nil)
        return &CheckoutResult{OrderID: input.OrderID, Status: "failed"}, err
    }

    // 5. Happy path
    workflow.ExecuteActivity(ctx, actRef.CheckoutConfirmInventory, input.OrderID).Get(ctx, nil)
    workflow.ExecuteActivity(ctx, actRef.CheckoutNotifyWarehouse, input.OrderID, input.ProductID).Get(ctx, nil)
    workflow.ExecuteActivity(ctx, actRef.CheckoutSendConfirmEmail, input.OrderID, input.UserID).Get(ctx, nil)

    // 6. Wait for delivery — 30-day timeout
    var delivery DeliverySignalPayload
    deliveryReceived := false

    dSel := workflow.NewSelector(ctx)
    dSel.AddReceive(workflow.GetSignalChannel(ctx, DeliverySignal),
        func(ch workflow.ReceiveChannel, _ bool) {
            ch.Receive(ctx, &delivery)
            deliveryReceived = true
        })
    dSel.AddFuture(workflow.NewTimer(ctx, DeliveryTimeoutDuration), func(_ workflow.Future) {})
    dSel.Select(ctx)

    if !deliveryReceived {
        workflow.ExecuteActivity(ctx, actRef.CheckoutAlertOps, input.OrderID, "delivery_timeout").Get(ctx, nil)
        return &CheckoutResult{OrderID: input.OrderID, Status: "paid"}, nil
    }

    workflow.ExecuteActivity(ctx, actRef.CheckoutSendReviewRequest, input.OrderID, input.UserID).Get(ctx, nil)
    return &CheckoutResult{OrderID: input.OrderID, Status: "delivered"}, nil
}

What’s happening

workflow.NewSelector — Temporal’s concurrency primitive. It races multiple futures/signals and unblocks when the first one fires. No goroutines, no channels. Deterministic replay safe.

workflow.NewTimer — A durable timer. If the worker crashes with 29 minutes elapsed, on restart Temporal replays history and the timer fires at the correct time. You don’t pay for the wait — the workflow is simply not scheduled.

Saga compensation — When ChargePayment fails (line 4), ReleaseInventory runs immediately in the error branch. This is the saga pattern: each forward step has a corresponding compensation that runs on failure.

nil actRef stub — Temporal resolves activities by method name via reflection. The var actRef Activities is a zero-value struct used only to get type-safe method references. The real *Activities instance (with injected dependencies) is what’s registered on the worker.

Activities with Injected Dependencies

Activities are methods on a struct, so you can inject real dependencies:

type Activities struct {
    Inventory InventoryService
    Payment   PaymentGateway
    Warehouse WarehouseService
    Notifier  NotifierService
    Log       logger.Logger
}

func (a *Activities) CheckoutReserveInventory(ctx context.Context, input CheckoutInput) error {
    ctx, span := otel.StartSpan(ctx, "CheckoutReserveInventory")
    defer span.End()

    a.Log.Info("reserving inventory", "order_id", input.OrderID, "product_id", input.ProductID)
    return a.Inventory.Reserve(ctx, input.OrderID, input.ProductID, 1)
}

func (a *Activities) CheckoutChargePayment(ctx context.Context, payment PaymentSignalPayload) error {
    ctx, span := otel.StartSpan(ctx, "CheckoutChargePayment")
    defer span.End()

    return a.Payment.Charge(ctx, payment.OrderID, payment.Amount, payment.TransactionID)
}

Activity naming convention — prefix every method with the module name (Checkout). Temporal registers activities globally on a worker. Without prefixes, ReserveInventory from orders and ReserveInventory from checkout would collide.

Worker Registration

Each module gets its own worker on its own task queue. This enables independent scaling:

// checkout-svc entrypoint
checkoutActs := &checkouttemporal.Activities{
    Inventory: checkoutfake.NewFakeInventoryService(log),
    Payment:   checkoutfake.NewFakePaymentGateway(log),
    Warehouse: checkoutfake.NewFakeWarehouseService(log),
    Notifier:  checkoutfake.NewFakeNotifier(log),
    Log:       log,
}

w := worker.New(temporalClient, checkouttemporal.TaskQueue, worker.Options{
    Interceptors: []interceptor.WorkerInterceptor{tracingInterceptor},
})
w.RegisterWorkflow(checkouttemporal.CheckoutWorkflow)
w.RegisterActivity(checkoutActs.CheckoutReserveInventory)
w.RegisterActivity(checkoutActs.CheckoutReleaseInventory)
w.RegisterActivity(checkoutActs.CheckoutChargePayment)
w.RegisterActivity(checkoutActs.CheckoutConfirmInventory)
w.RegisterActivity(checkoutActs.CheckoutNotifyWarehouse)
w.RegisterActivity(checkoutActs.CheckoutSendConfirmEmail)
w.RegisterActivity(checkoutActs.CheckoutSendTimeoutEmail)
w.RegisterActivity(checkoutActs.CheckoutSendReviewRequest)
w.RegisterActivity(checkoutActs.CheckoutAlertOps)

Task queues are simple strings. Each service declares its own:

// modules/orders/infrastructure/temporal/order_workflow.go
const TaskQueue = "order-task-queue"

// modules/refunds/infrastructure/temporal/refund_workflow.go
const TaskQueue = "refund-task-queue"

// modules/checkout/infrastructure/temporal/checkout_workflow.go
const TaskQueue = "checkout-task-queue"

Sending Signals from a Webhook

The payment webhook receives a callback from the payment gateway, verifies the HMAC signature, deduplicates via an idempotency store, then signals the running workflow:

func (h *CheckoutHandler) handlePaymentWebhook(c *fiber.Ctx) error {
    body := c.Body()

    // 1. Verify HMAC-SHA256 signature
    if !checkoutVerifyHMAC(body, c.Get("X-Signature"), h.webhookSecret) {
        return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{"error": "invalid signature"})
    }

    var payload checkoutdto.PaymentWebhookRequest
    json.Unmarshal(body, &payload)

    // 2. Idempotency: process each transactionID exactly once
    key := "webhook:payment:" + payload.TransactionID
    ok, err := h.idempotency.SetNX(c.UserContext(), key, 24*time.Hour)
    if !ok {
        return c.JSON(fiber.Map{"status": "already_processed"})
    }

    if payload.Status != "SUCCESS" {
        return c.JSON(fiber.Map{"status": "ignored"})
    }

    // 3. Signal the running workflow — unblocks workflow.Select
    if err := h.checkout.SignalPayment(c.UserContext(), payload); err != nil {
        if isNotFoundError(err) {
            // Workflow already timed out — log and report auto-refund intent
            return c.JSON(fiber.Map{"status": "refunded"})
        }
        return c.Status(500).JSON(fiber.Map{"error": err.Error()})
    }

    return c.JSON(fiber.Map{"status": "signaled"})
}

The service layer calls SignalWorkflow:

func (s *checkoutService) SignalPayment(ctx context.Context, payload dto.PaymentWebhookRequest) error {
    ctx, span := otel.StartSpan(ctx, "CheckoutService.SignalPayment")
    defer span.End()

    return s.temporal.SignalWorkflow(ctx, payload.OrderID, "", checkouttemporal.PaymentReceivedSignal,
        checkouttemporal.PaymentSignalPayload{
            OrderID:       payload.OrderID,
            Amount:        payload.Amount,
            TransactionID: payload.TransactionID,
            Method:        payload.Method,
        })
}

The payload.OrderID is used as the workflow ID — so you can signal the right workflow without knowing the run ID.

OpenTelemetry Integration

Every workflow execution produces a distributed trace across HTTP handler → service → Temporal worker → activities:

// Tracing interceptor wires OTel into every workflow and activity
tracingInterceptor, _ := temporalotel.NewTracingInterceptor(temporalotel.TracerOptions{})

temporalClient, _ := client.Dial(client.Options{
    HostPort:     cfg.TemporalHost,
    Interceptors: []interceptor.ClientInterceptor{tracingInterceptor},
})

w := worker.New(temporalClient, TaskQueue, worker.Options{
    Interceptors: []interceptor.WorkerInterceptor{tracingInterceptor},
})

The resulting trace in Grafana Tempo:

POST /api/v1/checkout
  └─ CheckoutService.StartCheckout
       ├─ CheckoutRepository.Save
       └─ Temporal: StartWorkflow
            └─ CheckoutWorkflow
                 ├─ CheckoutReserveInventory   (activity span)
                 ├─ [signal: payment-received]
                 ├─ CheckoutChargePayment      (activity span)
                 ├─ CheckoutConfirmInventory   (activity span)
                 ├─ CheckoutNotifyWarehouse    (activity span)
                 ├─ CheckoutSendConfirmEmail   (activity span)
                 ├─ [signal: delivery-confirmed]
                 └─ CheckoutSendReviewRequest  (activity span)

Microservices via CLI Subcommands

The entire project builds to a single binary with 5 subcommands — one all-in-one monolith and 4 independent services:

go run main.go server          # monolith, port :8080, all task queues
go run main.go orders-svc      # port :8081, order-task-queue only
go run main.go refunds-svc     # port :8082, refund-task-queue only
go run main.go shipments-svc   # port :8083, shipment-task-queue only
go run main.go checkout-svc    # port :8084, checkout-task-queue only

Each service starts its own Temporal worker + Fiber HTTP server. No service-to-service HTTP calls — Temporal is the sole integration point between modules.

A shared config package reads from CLI flags or env vars:

// pkg/config/config.go
type ServiceConfig struct {
    ServiceName   string
    TemporalHost  string  // TEMPORAL_HOST, default: localhost:7233
    OTelEndpoint  string  // OTEL_ENDPOINT, default: localhost:4317
    ListenAddr    string  // LISTEN_ADDR, default: :8080
    WebhookSecret string  // WEBHOOK_SECRET, default: dev-secret
}

Workflow State Machine

The checkout entity tracks status through the workflow lifecycle:

pending
  → reserved    (ReserveInventory succeeds)
  → paid        (ChargePayment succeeds)
  → delivered   (delivery-confirmed signal received)
  → timeout     (30-min payment timer fired)
  → failed      (ChargePayment failed → ReleaseInventory ran)
  → paid        (delivery 30-day timer fired → AlertOps)

Testing Workflows

Temporal provides a testsuite package for unit testing workflows without a running server:

func TestCheckoutWorkflow_PaymentTimeout(t *testing.T) {
    suite := &testsuite.WorkflowTestSuite{}
    env := suite.NewTestWorkflowEnvironment()

    env.RegisterWorkflow(CheckoutWorkflow)
    env.RegisterActivity(&Activities{})

    // Fast-forward the 30-minute timer immediately
    env.SetTestTimeout(time.Second)

    env.OnActivity("CheckoutReserveInventory", mock.Anything, mock.Anything).Return(nil)
    env.OnActivity("CheckoutReleaseInventory", mock.Anything, mock.Anything, mock.Anything).Return(nil)
    env.OnActivity("CheckoutSendTimeoutEmail", mock.Anything, mock.Anything, mock.Anything).Return(nil)

    env.ExecuteWorkflow(CheckoutWorkflow, CheckoutInput{
        OrderID: "o-1", UserID: "u-1", ProductID: "p-1", Amount: 99.99,
    })

    require.True(t, env.IsWorkflowCompleted())
    require.NoError(t, env.GetWorkflowError())

    var result CheckoutResult
    env.GetWorkflowResult(&result)
    assert.Equal(t, "timeout", result.Status)
}

The test environment runs timers synchronously, so the 30-minute timer fires instantly in tests.

When to Use Temporal

Good fit:

• Multi-step business transactions (checkout, onboarding, order fulfillment)
• Long-running processes that span minutes, hours, or days
• Workflows needing compensation (saga pattern)
• Processes that wait for external events (webhooks, human approval)
• Anywhere you’d otherwise poll a status column

Not a good fit:

• Simple request/response APIs with no async steps
• Sub-millisecond latency requirements (Temporal adds ~10ms per activity)
• Pure data pipelines (batch ETL — use Spark/Flink)

Running Locally

# Start Temporal server + Postgres + Grafana Tempo
make infra-up

# Run the checkout service
make run-checkout

# Start a checkout
curl -X POST http://localhost:8084/api/v1/checkout \
  -H "Content-Type: application/json" \
  -d '{"order_id":"o-1","user_id":"u-1","product_id":"p-1","amount":99.99}'

# Send signed payment webhook
BODY='{"order_id":"o-1","amount":99.99,"transaction_id":"txn-1","method":"card","status":"SUCCESS"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "dev-secret" | awk '{print $2}')
curl -X POST http://localhost:8084/api/v1/webhooks/payment \
  -H "Content-Type: application/json" -H "X-Signature: $SIG" -d "$BODY"

# Signal delivery
curl -X POST "http://localhost:8084/api/v1/checkout/o-1/deliver?tracking=TRACK-123"

Open http://localhost:8088 to see the full workflow execution history in the Temporal UI. Open http://localhost:3000 for distributed traces in Grafana Tempo.

Key Takeaways

• Temporal replaces state machines + queues + retry logic — write plain Go functions, get durability for free
• workflow.Select races signals against timers deterministically — no goroutines needed
• Saga compensation is just an if err != nil branch that calls the undo activity
• One task queue per service enables independent worker scaling and clean service boundaries
• The nil activity stub pattern (var actRef Activities) gives type-safe method references without constructing a real instance in the workflow
• OTel tracing interceptor on both the client and worker automatically connects HTTP traces to workflow/activity spans