Temporal TransferWorkflow in Go: Internal Moves, External Holds, and Bank UNKNOWN

In the introduction post, Temporal was a set of primitives: workflows, activities, retries, task queues. This post is the production shape those primitives take in a real wallet system — and why we stopped writing the saga by hand in Kafka consumers.

The core workflow is small enough to fit in one file and strict enough that the failure modes are explicit:

• INTERNAL transfer: prepare settlement → Wallet.Move → mark terminal
• EXTERNAL transfer: prepare → Wallet.Hold → Bank.Submit → settle, release, or keep the hold while the bank is UNKNOWN

The important architectural move is not “use Temporal”. It is take money movement out of a custom Go orchestration path and put the saga under one durable controller.

Before Temporal: Custom Saga Orchestration in Go

We already had a saga. It just did not look like one product file.

The old controller was a Kafka-driven Go workflow spread across several processes:

POST /transfers
  → transfers(PENDING) + outbox(transfer.requested)
iris / outbox drain
  → Kafka transfer.requested
cmd/consumer (wallet-processor)
  → INTERNAL: ExecuteInternalTransfer(...)
  → EXTERNAL: ReserveExternalTransfer(...)
              + outbox(transfer.provider.requested)
provider consumer
  → call stub-provider / bank HTTP
  → SettleExternalTransfer(SUCCESS|FAILURE)
retry consumers (6s / 30s / 5m)
  → republish failed work after delay
DLQ
  → human archaeology

How the custom INTERNAL path worked

For internal transfers, the consumer called a repository method that tried to do the whole happy path in one Postgres transaction:

// Legacy path (conceptually)
err := transferRepo.ExecuteInternalTransfer(ctx, transferID, fxService)

Inside that transaction the code would:

1. load the PENDING transfer
2. quote FX + fee
3. debit source / credit destination / write ledger (+ optional fee)
4. mark the transfer terminal
5. write transfer.completed or transfer.failed into the outbox

That is attractive when Transfer and Wallet share one database. It is also a trap:

• the “workflow” is buried in repository SQL and consumer branching
• crash windows still exist around Kafka ack / inbox / outbox publish
• retry semantics depend on which consumer retried which topic
• you cannot pause, inspect, or continue a half-finished process except by re-reading rows and guessing

How the custom EXTERNAL path worked

External settlement was a two-stage custom saga:

1) ReserveExternalTransfer
   available → hold
   PENDING  → RESERVED
   outbox(transfer.provider.requested)

2) provider consumer
   HTTP submit to stub-provider
   on response:
     SettleExternalTransfer(SUCCESS) → settle hold + terminal success
     SettleExternalTransfer(FAILURE) → release hold + terminal failure

The state machine lived in:

• transfer status columns (PENDING, RESERVED, PROCESSING, SUCCEEDED, FAILED)
• outbox event types
• provider consumer code
• delayed-retry topic names
• operator knowledge about which topic meant “safe to retry”

That is orchestration. It is just orchestration by side effects.

What the custom controller had to reinvent

Once you own the saga yourself, you also own every platform concern around it:

The worst part was not writing the first version. It was explaining the tenth failure:

“Is this transfer waiting on provider, stuck in retry-30s, already reserved, or terminal with a missing notification?”

Four systems had partial answers. None had the full story.

The real failure modes we kept hitting

1. Controller and side effects lived in the same process mental model

The Kafka consumer both decided what happens next and performed money movement. A bad deploy or poison message was not just a routing bug — it was a money bug.

2. EXTERNAL timeout was under-modeled

HTTP provider timeouts look like errors. In money systems they are often UNKNOWN. The custom path tended to escalate retries as if the bank had not accepted the payment. That pushes you toward unsafe release or endless topic churn.

3. Retry policy was infrastructure, not business code

Whether a step retried in 6 seconds or 5 minutes depended on topic topology, not on a typed policy next to the step. New engineers had to learn the broker layout before they could reason about transfer lifecycle.

4. Canarying a new orchestration path was awkward

We could not cleanly say “INTERNAL goes through the new controller, EXTERNAL stays on the old one” without forking consumer logic and fearing double-consume. Kafka is pub/sub: a second consumer group on transfer.requested receives the same messages unless both sides filter perfectly.

5. Local transactions hid the service boundary we wanted

ExecuteInternalTransfer could touch transfer tables and wallet tables in one pgx.Tx. That is correct for a modular monolith. It becomes impossible the moment Wallet is a separate service with its own database. The custom saga was optimized for today’s package layout, not tomorrow’s service split.

Migration Strategy: Temporal Becomes the Controller

We did not flip a global switch on day one. The migration was deliberate:

P4  Starter bridge exists, but TemporalRouted = []
    → all traffic still legacy

P5  Route INTERNAL → TransferWorkflow
    → EXTERNAL still ExecuteInternal/Reserve path

P6  Route EXTERNAL → TransferWorkflow
    → legacy external path becomes dead for new traffic

P7  Delete provider consumer, stub-provider money path,
    legacy branches, transfer retry-topic orchestration
    → consumer remains only as Kafka→Temporal bridge

Important design choice: keep one consumer group (wallet-processor).

We considered a dedicated transfer-starter service. That would have been cleaner on paper and dangerous in practice. Two groups on transfer.requested means double start unless both implementations are perfect during the whole canary. Instead we extended the existing processor:

execute(transferID):
  if type in TemporalRouted:
      StartWorkflow(transfer-{id})
  else:
      legacy ExecuteInternalTransfer / ReserveExternalTransfer

During the canary, Temporal was the controller for some types while the custom saga remained the controller for others. After cutover, the legacy branch died and the consumer shrank to:

fetch → inbox dedup → StartWorkflow → commit

Kafka stayed. The custom workflow engine left.

Why the Old Shape Breaks (and What Temporal Replaces)

The old shape fails for process reasons, not syntax reasons:

1. Money and status drift across retries and multi-step external settlement
2. Bank timeouts are not failures — “no response” is not “rejected”
3. Retry policy lives in tribal knowledge — topics and consumer code instead of step policies
4. Ops cannot see the process — only topic lag and a status column
5. Service split is blocked by single-DB repository sagas

Temporal replaces the custom controller responsibilities:

We kept Kafka for durable fan-out from the outbox and for notifications. We stopped using it as the brain of the transfer.

Target Architecture: Temporal as Controller

HTTP Transfer API
  → transfers(PENDING) + outbox(transfer.requested)
iris CDC
  → Kafka topic transfer.requested
consumer (bridge only)
  → Temporal StartWorkflow(WorkflowID=transfer-{id})
transfer-worker
  → TransferWorkflow
      activities:
        LoadTransfer / Prepare*
        Wallet gRPC (Move / Hold / SettleHold / ReleaseHold)
        Bank gRPC (Submit / Query)
        MarkTerminal (status + outbox)
iris
  → transfer.completed / transfer.failed → notification

Responsibilities become sharp:

Temporal is the orchestrator. Kafka is the delivery bus into Temporal, then the delivery bus out to notifications.

Starting the Workflow Idempotently

The consumer is intentionally boring:

func (p *Processor) startTransferWorkflow(ctx context.Context, transferID uuid.UUID) error {
    workflowID := fmt.Sprintf("transfer-%s", transferID.String())
    _, err := p.temporal.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
        ID:                    workflowID,
        TaskQueue:             p.temporalQueue,
        WorkflowIDReusePolicy: enumspb.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE,
        WorkflowExecutionErrorWhenAlreadyStarted: true,
    }, worker.TransferWorkflow, worker.TransferWorkflowInput{
        TransferID: transferID.String(),
    })
    if err != nil {
        var alreadyStarted *serviceerror.WorkflowExecutionAlreadyStarted
        if errors.As(err, &alreadyStarted) {
            return nil // idempotent start
        }
        return err
    }
    return nil
}

Two layers of start idempotency:

1. Inbox table — (consumer_group, transfer_id) so Kafka redelivery does not spam starts
2. Workflow ID — transfer-{id} so a race still collapses to one run

If Temporal is temporarily down, the bridge escalates through retry topics / DLQ. It never moves money while “just retrying the message”.

TransferWorkflow Skeleton

func TransferWorkflow(ctx workflow.Context, input TransferWorkflowInput) error {
    actx := workflow.WithActivityOptions(ctx, defaultActivityOptions())
    var activities *Activities

    var loaded LoadTransferResult
    if err := workflow.ExecuteActivity(actx, activities.LoadTransfer, transferID).Get(ctx, &loaded); err != nil {
        return err
    }
    if loaded.AlreadyTerminal {
        return nil
    }

    switch loaded.TransferType {
    case "INTERNAL":
        return runInternalTransfer(ctx, actx, activities, transferID)
    case "EXTERNAL":
        return runExternalTransfer(ctx, actx, activities, transferID)
    default:
        // terminalize unsupported type so it does not stick PENDING forever
        _ = workflow.ExecuteActivity(terminalActivityContext(ctx), activities.MarkTerminal, MarkTerminalInput{
            TransferID: transferID,
            Succeeded:  false,
            Reason:     "PROVIDER_REJECTED",
        }).Get(ctx, nil)
        return temporal.NewNonRetryableApplicationError(
            fmt.Sprintf("unsupported transfer type %q", loaded.TransferType),
            BusinessErrorType,
            nil,
        )
    }
}

The workflow input is only the transfer ID. Everything authoritative is loaded from the transfer row. That keeps the starter bridge dumb and avoids shipping a stale snapshot through Kafka.

INTERNAL Path: Prepare → Move → Terminal

Internal transfers can be same-currency or cross-currency. Settlement amounts are not trusted from the client. The prepare activity quotes FX, freezes a settlement snapshot, and advances PENDING → PROCESSING.

func runInternalTransfer(
    ctx workflow.Context,
    actx workflow.Context,
    activities *Activities,
    transferID uuid.UUID,
) error {
    tctx := terminalActivityContext(ctx)

    var prepared PrepareInternalMoveResult
    if err := workflow.ExecuteActivity(actx, activities.PrepareInternalMove, PrepareInternalMoveInput{
        TransferID: transferID,
    }).Get(ctx, &prepared); err != nil {
        return markFailedIfBusiness(ctx, tctx, activities, transferID, err)
    }
    if prepared.AlreadyTerminal {
        return nil
    }

    if err := workflow.ExecuteActivity(actx, activities.WalletMove, WalletMoveInput{
        TransferID:          transferID,
        Operation:           "MOVE",
        FromAccountRef:      prepared.FromAccountRef,
        ToAccountRef:        prepared.ToAccountRef,
        SourceAmount:        prepared.SourceAmount,
        SourceCurrency:      prepared.SourceCurrency,
        DestinationAmount:   prepared.DestinationAmount,
        DestinationCurrency: prepared.DestinationCurrency,
        FeeAmount:           prepared.FeeAmount,
        FeeCurrency:         prepared.FeeCurrency,
    }).Get(ctx, nil); err != nil {
        return markFailedIfBusiness(ctx, tctx, activities, transferID, err)
    }

    return workflow.ExecuteActivity(tctx, activities.MarkTerminal, MarkTerminalInput{
        TransferID: transferID,
        Succeeded:  true,
    }).Get(ctx, nil)
}

Why prepare is separate from move

Prepare does three things money movement should not re-decide later:

1. validate both accounts are active
2. quote source/destination/fee amounts
3. freeze those amounts on the transfer row

If Wallet.Move retries, it reuses the frozen snapshot. Rates do not drift between attempts.

Why MarkTerminal has a longer retry budget

Money lives in Wallet. Status lives in Transfer. They are different transactions on purpose so services can split later.

That creates a dangerous window:

Wallet.Move committed
MarkTerminal fails (DB blip)

If the workflow gives up, you have moved money and a non-terminal transfer. So MarkTerminal uses a much longer schedule-to-close and unlimited attempts until that budget is exhausted:

func terminalActivityContext(ctx workflow.Context) workflow.Context {
    return workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout:    30 * time.Second,
        ScheduleToCloseTimeout: 24 * time.Hour,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumInterval:    time.Minute,
            MaximumAttempts:    0, // unlimited until schedule-to-close
            NonRetryableErrorTypes: []string{BusinessErrorType},
        },
    })
}

After money moves, status must converge.

EXTERNAL Path: Hold, Then Let the Bank Decide

External transfers never convert currency in this design. Source account currency must equal transaction currency. Then:

PrepareExternalHold
  → Wallet.Hold
  → Bank.Submit
      SUCCESS  → SettleHold → MarkTerminal(SUCCEEDED)
      FAILURE  → ReleaseHold → MarkTerminal(FAILED)
      UNKNOWN  → keep hold, poll Bank.Query

func runExternalTransfer(...) error {
    // prepare + hold omitted for brevity

    var bank BankResult
    if err := workflow.ExecuteActivity(actx, activities.BankSubmit, BankSubmitInput{
        TransferID: transferID,
        Amount:     prepared.Amount,
        Currency:   prepared.Currency,
    }).Get(ctx, &bank); err != nil {
        // transient transport error: retry activity, hold remains
        return err
    }

    return settleExternalFromBank(ctx, actx, tctx, activities, transferID, prepared, bank)
}

The UNKNOWN rule

Bank timeout is mapped to an outcome, not a thrown error:

func mapBankError(err error) (BankResult, error) {
    st, ok := status.FromError(err)
    if ok && st.Code() == codes.DeadlineExceeded {
        return BankResult{Outcome: "UNKNOWN", Reason: "TIMEOUT"}, nil
    }
    return BankResult{}, err
}

Why? Because a timeout means we do not know. The bank may have accepted the payment. Auto-releasing the hold can free funds that already left. Auto-settling can debit a payment that never happened.

So the workflow polls:

func pollBankUntilKnown(...) error {
    start := workflow.Now(ctx)
    alerted := false

    for {
        elapsed := workflow.Now(ctx).Sub(start)
        if elapsed >= 24*time.Hour {
            // retryable workflow error — hold still retained for manual recon
            return fmt.Errorf("bank outcome still UNKNOWN after %s", elapsed)
        }
        if !alerted && elapsed >= 15*time.Minute {
            logger.Error("bank UNKNOWN past alert threshold; hold retained")
            alerted = true
        }

        _ = workflow.Sleep(ctx, 30*time.Second)

        var bank BankResult
        if err := workflow.ExecuteActivity(actx, activities.BankQuery, transferID).Get(ctx, &bank); err != nil {
            continue // keep polling
        }
        if bank.Outcome == "SUCCESS" || bank.Outcome == "FAILURE" {
            return settleExternalFromBank(ctx, actx, tctx, activities, transferID, prepared, bank)
        }
    }
}

This is the saga rule people under-specify:

UNKNOWN is not FAILURE. Holds are retained until the bank is known or a human reconciles.

Business Errors vs Transient Errors

Activities translate permanent wallet failures into non-retryable Temporal application errors:

const BusinessErrorType = "TRANSFER_BUSINESS"

func businessError(reason string) error {
    return temporal.NewNonRetryableApplicationError(reason, BusinessErrorType, nil)
}

func mapWalletMoveError(err error) error {
    st, ok := status.FromError(err)
    if !ok {
        return err
    }
    switch st.Code() {
    case codes.FailedPrecondition:
        if strings.Contains(st.Message(), "insufficient funds") {
            return businessError("INSUFFICIENT_FUNDS")
        }
        return businessError("ACCOUNT_NOT_ACTIVE")
    case codes.Unavailable, codes.DeadlineExceeded, codes.ResourceExhausted, codes.Aborted:
        return err // retryable
    default:
        return err
    }
}

The workflow then terminalizes only permanent failures:

func markFailedIfBusiness(...) error {
    if !isBusinessError(err) {
        return err // Temporal retries
    }
    return workflow.ExecuteActivity(tctx, activities.MarkTerminal, MarkTerminalInput{
        TransferID: transferID,
        Succeeded:  false,
        Reason:     businessReason(err),
    }).Get(ctx, nil)
}

Examples:

Idempotency Is a Stack, Not a Flag

Temporal durability does not replace idempotency. It multiplies the need for it.

Wallet RPCs always carry transfer_id + operation. A redelivered Hold or Move with the same pair is a no-op that returns current balances. That is what makes activity retry safe.

Worker Wiring

The worker process is a normal Go service with a Temporal poller and a health endpoint:

activities := transferworker.NewActivities(
    walletClient,
    bankClient,
    fxService,
    transferRepo,
    accountRepo,
)

w := worker.New(temporalClient, cfg.Temporal.TaskQueue, worker.Options{})
w.RegisterWorkflow(transferworker.TransferWorkflow)
w.RegisterActivity(activities)

Money activities are thin gRPC adapters. Prepare/MarkTerminal talk to transfer-owned tables in-process. That keeps the workflow free of infrastructure details while still respecting module ownership.

Testing the Saga Paths

Temporal’s testsuite lets us pin the failure matrix without bank or Kafka:

func (s *TransferWorkflowTestSuite) TestExternalUnknownThenSuccess() {
    transferID := uuid.New()
    a := s.registerExternalBase(transferID)

    s.env.OnActivity(a.BankSubmit, mock.Anything, mock.Anything).Return(
        worker.BankResult{Outcome: worker.BankOutcomeUnknown, Reason: "TIMEOUT"}, nil,
    )
    s.env.OnActivity(a.BankQuery, mock.Anything, transferID).Return(
        worker.BankResult{Outcome: worker.BankOutcomeUnknown}, nil,
    ).Once()
    s.env.OnActivity(a.BankQuery, mock.Anything, transferID).Return(
        worker.BankResult{Outcome: worker.BankOutcomeSuccess, ReferenceID: "late-ref"}, nil,
    ).Once()
    s.env.OnActivity(a.WalletSettleHold, mock.Anything, mock.Anything).Return(worker.WalletHoldResult{}, nil)
    s.env.OnActivity(a.MarkTerminal, mock.Anything, worker.MarkTerminalInput{
        TransferID:  transferID,
        Succeeded:   true,
        ReferenceID: "late-ref",
    }).Return(nil)

    s.env.ExecuteWorkflow(worker.TransferWorkflow, worker.TransferWorkflowInput{
        TransferID: transferID.String(),
    })
    s.True(s.env.IsWorkflowCompleted())
    s.NoError(s.env.GetWorkflowError())
}

Other cases worth locking in:

• internal same-currency success
• internal cross-currency success
• insufficient funds → FAILED, no success path
• FX unavailable during prepare → FAILED before money moves
• already terminal → no-op
• MarkTerminal flakes after Move → retries, Move not re-called as a new business decision
• EXTERNAL UNKNOWN for multiple polls → no release/settle until known
• currency mismatch on EXTERNAL → fail before hold

If a money workflow is not tested against its fault matrix, it is only a happy-path script.

Sequence: EXTERNAL Success vs UNKNOWN

sequenceDiagram
    participant API as Transfer API
    participant BR as consumer bridge
    participant T as Temporal
    participant W as transfer-worker
    participant WG as wallet-grpc
    participant BG as bank-grpc

    API->>BR: transfer.requested
    BR->>T: StartWorkflow(transfer-{id})
    T->>W: TransferWorkflow EXTERNAL
    W->>WG: Hold
    W->>BG: Submit

    alt SUCCESS
        BG-->>W: SUCCESS
        W->>WG: SettleHold
        W->>W: MarkTerminal SUCCEEDED
    else FAILURE
        BG-->>W: FAILURE
        W->>WG: ReleaseHold
        W->>W: MarkTerminal FAILED
    else UNKNOWN
        BG-->>W: UNKNOWN
        loop poll, hold retained
            W->>BG: Query
        end
    end

Custom Controller vs Temporal Controller

Side by side, the control plane change looks like this:

BEFORE (custom Go saga)
  consumer decides next step
  repo methods move money + status
  provider consumer continues external saga
  retry topics are the timer service
  status column + logs are the debugger

AFTER (Temporal controller)
  consumer only starts workflow
  TransferWorkflow decides next step
  activities move money / call bank / mark terminal
  Temporal timers/retries are the timer service
  workflow history is the debugger

What improved immediately:

1. One readable controller — transfer_workflow.go instead of consumer + provider consumer + retry consumers + repo saga methods
2. Safer canary — route by transfer_type inside one consumer group, no double-consume second group
3. Better UNKNOWN handling — timeout is an outcome with hold retention, not “retry topic soup”
4. Service-split ready — money over Wallet gRPC, status over Transfer activities, no shared pgx.Tx required for the saga
5. Supportability — open Temporal UI, see exact activity attempts and bank poll loop

What we still keep from the old world:

• outbox + CDC for reliable event emission
• Kafka for notification fan-out
• inbox dedup at the bridge
• wallet operation guards for money idempotency

Temporal did not replace messaging. It replaced the homemade workflow engine that had grown inside messaging.

What We Intentionally Did Not Put in the Workflow

A few non-goals kept the design honest:

1. No payment webhook signals here — bank interaction is submit/query, not a checkout-style signal race
2. No money movement in the Kafka consumer — bridge only
3. No auto-release on timeout — ops/reconciliation beats silent fund release
4. No client-supplied settlement amounts — prepare freezes server-side quotes
5. No giant workflow input DTO — load the transfer by ID inside the first activity
6. No second consumer group during migration — route in-place to avoid double start

Temporal makes long compensation graphs easy to write. It does not make over-modeling free.

Operational Checklist

Before calling this production-ready, verify:

• Workflow IDs are deterministic per transfer
• Wallet operations are idempotent on (transfer_id, operation)
• Business errors are non-retryable application errors
• MarkTerminal retries hard after money movement
• Bank timeouts become UNKNOWN, not FAILED
• UNKNOWN path never settle/release automatically
• consumer DLQ covers start poison, not money poison
• legacy provider/retry orchestration paths are decommissioned after cutover
• Temporal UI is reachable for support/debug
• fault-matrix workflow tests cover success, business fail, and UNKNOWN

Key Takeaways

• We already had a saga before Temporal — it was just a custom Go/Kafka controller
• Temporal becomes the controller; Kafka becomes the transport
• Migrate by routing transfer types through one consumer group, then delete legacy branches
• Keep the Kafka consumer as a starter bridge with inbox dedup
• Split INTERNAL and EXTERNAL paths explicitly — different money shapes, different bank risk
• Freeze settlement in prepare; do not re-quote on every activity retry
• Treat money and status as separate transactions, then force status convergence with long MarkTerminal retries
• Model bank timeout as UNKNOWN and retain the hold
• Stack idempotency at API, inbox, workflow ID, wallet guards, and terminal status

Closing

Temporal did not invent the transfer saga for us. The custom path already had INTERNAL execute, EXTERNAL reserve/settle, provider callbacks, and delayed retries.

What Temporal changed is who owns the control plane:

before: consumer + provider consumer + retry topics + repo methods
after:  TransferWorkflow + activities + Temporal history

The workflow file is now the product spec:

INTERNAL: prepare → move → terminal
EXTERNAL: prepare → hold → bank → settle | release | poll forever carefully

If you are migrating an existing wallet pipeline, do not start by rewriting every service. Start by making the consumer boring and the saga explicit. Route one transfer type first. Prove the fault matrix. Then delete the homemade controller.

Once TransferWorkflow owns the lifecycle, the rest of the system gets simpler around it: Wallet becomes a money API, Bank becomes an outcome API, and Kafka goes back to being a bus instead of a half-finished workflow engine.

Next useful extensions from this foundation: child workflows per provider, cancellation/refund workflows with their own IDs, and search attributes so support can find open UNKNOWN holds without SQL archaeology.