Introduction to Temporal: Durable Workflows in Go Without the State Machine Spaghetti

Most backend systems eventually grow a half-finished workflow engine.

It starts innocently: a transfer is created as PENDING, a Kafka consumer picks it up, money moves in one service, status flips in another, a retry topic covers timeouts, a DLQ catches poison messages, and a cron job cleans up the leftovers. Six months later nobody can answer a simple question: where is this transfer stuck, and what is allowed to happen next?

Temporal is a better answer for that class of problem. You write ordinary Go functions. Temporal makes them durable, retryable, and observable.

This post is the foundation. The next one applies the same ideas to a real money-movement saga: TransferWorkflow in a wallet system.

The Problem Temporal Solves

A multi-step business transaction is hard for boring reasons:

1. Any step can fail — network blip, insufficient funds, bank timeout
2. The process can die mid-way — deploy, OOM, node loss
3. Some steps are not safe to re-run blindly — double debit is worse than a crash
4. Some steps take a long time — bank confirmation, human approval, webhook callback
5. You need a single story — not five logs in five services

Common alternatives:

Temporal is not “another queue”. It is a durable execution platform: the workflow code is the source of truth for the business process.

Core Mental Model

Think in four pieces:

Client / API
   │  StartWorkflow / SignalWorkflow / QueryWorkflow
   ▼
Temporal Server
   │  event history + timers + task dispatch
   ▼
Worker
   │  polls a task queue
   ├── Workflow code   (orchestration, deterministic)
   └── Activity code   (side effects: DB, gRPC, HTTP, email)

Workflow

A workflow is pure orchestration:

• call activities in order
• branch on results
• sleep / wait for signals
• compensate on failure

It must be deterministic. Temporal recovers state by replaying the event history. If your workflow code does something non-deterministic on replay (time.Now(), random UUID, unguarded goroutines, direct DB calls), history diverges and the run breaks.

Rule of thumb: side effects belong in activities.

Activity

An activity is a single side-effecting step:

• call Wallet gRPC
• write a DB row
• submit a bank transfer
• send an email

Activities can fail. Temporal retries them with a policy you control.

Worker

A worker is a process that:

1. connects to Temporal
2. polls a task queue
3. executes registered workflows and activities

You scale workers independently of the Temporal server. More load on transfers? Run more transfer workers.

Task queue

A task queue is just a string, for example transfer-task-queue. It is the routing key between “work that needs to run” and “workers that know how to run it”.

A Minimal Workflow in Go

package transfer

import (
    "time"

    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

type TransferInput struct {
    TransferID string
    Amount     string
    Currency   string
}

func TransferWorkflow(ctx workflow.Context, input TransferInput) error {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumInterval:    30 * time.Second,
            MaximumAttempts:    10,
            NonRetryableErrorTypes: []string{
                "TRANSFER_BUSINESS",
            },
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var acts *Activities // nil stub — Temporal resolves methods by name

    if err := workflow.ExecuteActivity(ctx, acts.Prepare, input).Get(ctx, nil); err != nil {
        return err
    }
    if err := workflow.ExecuteActivity(ctx, acts.MoveMoney, input).Get(ctx, nil); err != nil {
        return err
    }
    return workflow.ExecuteActivity(ctx, acts.MarkSucceeded, input.TransferID).Get(ctx, nil)
}

That reads like a script. Under the hood Temporal records each completed activity in history. If the worker dies after MoveMoney succeeds but before MarkSucceeded finishes, a new worker replays the history, skips completed steps, and continues from the unfinished one.

Activities Carry Dependencies

Workflows should stay thin. Activities hold the real clients:

type Activities struct {
    Wallet WalletClient
    Bank   BankClient
    DB     TransferRepository
}

func (a *Activities) MoveMoney(ctx context.Context, input TransferInput) error {
    return a.Wallet.Move(ctx, input.TransferID, input.Amount, input.Currency)
}

Register the concrete instance on the worker:

w := worker.New(temporalClient, "transfer-task-queue", worker.Options{})
w.RegisterWorkflow(TransferWorkflow)
w.RegisterActivity(activities)

The var acts *Activities inside the workflow is only a typed method reference. The worker-registered instance is what actually runs.

Retries: Transient vs Business Failures

Not every error should retry.

• Transient: network timeout, Unavailable, deadlocks, temporary DB blips
• Business / permanent: insufficient funds, inactive account, unsupported currency

Temporal models this with application errors:

func businessError(reason string) error {
    return temporal.NewNonRetryableApplicationError(
        reason,
        "TRANSFER_BUSINESS",
        nil,
    )
}

Put TRANSFER_BUSINESS in NonRetryableErrorTypes. Otherwise Temporal will hammer a permanent condition for the full retry budget.

This distinction matters more than almost any other Temporal design choice in money systems.

Orchestration vs Choreography

Choreography

Transfer API → transfer.requested
Wallet consumer → moves money → money.moved
Ledger consumer → writes entries → ledger.written
Notify consumer → sends push

Each service is “simple”. The process is not. Compensation becomes tribal knowledge. Tracing requires correlation IDs and optimism.

Orchestration with Temporal

workflow.ExecuteActivity(ctx, acts.Prepare, input)
workflow.ExecuteActivity(ctx, acts.MoveMoney, input)
workflow.ExecuteActivity(ctx, acts.MarkSucceeded, input.TransferID)

One function owns the sequence. The Temporal UI shows every attempt, timer, and failure. New engineers read the workflow instead of reverse-engineering topic graphs.

Choreography is still useful for fan-out notifications and integration events. It is a weak place to hide the core money lifecycle.

Durability Features You Actually Use

1. Automatic activity retries

Configure once. Stop rewriting the same exponential backoff helper.

2. Durable timers

workflow.Sleep(ctx, 30*time.Second)
// or
workflow.NewTimer(ctx, 15*time.Minute)

If the process restarts, the timer still fires at the correct time. The worker does not sit blocked burning CPU for 15 minutes either — Temporal parks the workflow.

3. Signals

External events can wake a workflow:

signalChan := workflow.GetSignalChannel(ctx, "payment-received")
signalChan.Receive(ctx, &payment)

Useful for webhooks, human approval, or async provider callbacks.

4. Full execution history

Every schedule, start, completion, failure, and signal is recorded. That is your audit trail for free.

Testing Without a Cluster

Temporal’s Go SDK ships a test environment:

func TestTransferWorkflow_Success(t *testing.T) {
    var suite testsuite.WorkflowTestSuite
    env := suite.NewTestWorkflowEnvironment()

    var a *Activities
    env.RegisterWorkflow(TransferWorkflow)
    env.RegisterActivity(a.Prepare)
    env.RegisterActivity(a.MoveMoney)
    env.RegisterActivity(a.MarkSucceeded)

    env.OnActivity(a.Prepare, mock.Anything, mock.Anything).Return(nil)
    env.OnActivity(a.MoveMoney, mock.Anything, mock.Anything).Return(nil)
    env.OnActivity(a.MarkSucceeded, mock.Anything, mock.Anything).Return(nil)

    env.ExecuteWorkflow(TransferWorkflow, TransferInput{
        TransferID: "t-1",
        Amount:     "100",
        Currency:   "USD",
    })

    require.True(t, env.IsWorkflowCompleted())
    require.NoError(t, env.GetWorkflowError())
}

Timers advance in test time. You can assert which activities ran, with which inputs, after which failures.

When Temporal Is a Good Fit

Use it when:

• the business process has multiple steps with different failure modes
• retries and compensation are part of correctness, not just ops polish
• the flow can last seconds to days
• you need a single operational view of “what happened”
• you are about to invent workflow state in Postgres + Kafka

Skip it when:

• the API is a single short DB transaction
• you need ultra-low latency request/response with no async steps
• the team cannot run another platform dependency yet
• the problem is pure batch analytics

Temporal adds operational surface area. It pays rent when process durability is the product risk.

A Production-Shaped Skeleton

In a real service, the split usually looks like this:

POST /transfers
  → write PENDING transfer + outbox event
CDC / consumer
  → StartWorkflow(WorkflowID = transfer-{id})
transfer-worker
  → TransferWorkflow
      → activities: prepare, wallet, bank, mark terminal
UI / support
  → inspect history in Temporal Web

Two details matter immediately:

1. Deterministic workflow IDs — transfer-{uuid} makes start idempotent
2. Idempotent activities — wallet operations key on (transfer_id, operation) so retries are safe

Without those, durability just helps you fail more confidently.

Key Takeaways

• Temporal lets you write multi-step business logic as Go code while the platform handles retries, recovery, and history
• Workflows orchestrate; activities perform side effects
• Separate business errors from transient errors or retries will hurt you
• Orchestration is easier to reason about than a web of consumers for core transactions
• Deterministic workflow IDs + idempotent activities are mandatory in money systems
• Test workflows with testsuite before you ever open the Temporal UI

What’s Next

Concepts are cheap. Money movement is not.

In the next post, we take these primitives into a real wallet backend and implement TransferWorkflow:

• INTERNAL path: quote → move → mark terminal
• EXTERNAL path: hold → bank submit → settle / release / poll on UNKNOWN
• why money and status live in different transactions
• how the Kafka consumer becomes only a Temporal starter, not a money mover

If you already know Temporal and want the production saga, skip ahead to that post. If you are still mapping Temporal onto your own architecture, this model is enough to start: one workflow per business transaction, activities for every side effect, and no hidden state machine in the database.