Continue the deploy workflow as new to bound its history

[emmanuelmaduwuba-lyft]

Why

The Deploy workflow only exits when its queue is empty. A queue that is locked with merged revisions waiting keeps the workflow alive indefinitely, and each idle hour adds a timer and a workflow task to its history, roughly 125 events a day. Two things go wrong as it grows:

1. Past Temporal's hard history limit, the server terminates the workflow. The queue and lock live only in workflow memory (queue/worker.go: "we don't persist lock state anywhere"), so queued deploys are dropped without any notice.
2. A large history can stop a workflow from making progress. After a worker restart, replaying it can exceed the SDK's 1-second deadlock detection, and every workflow task then fails.

What changes

The workflow continues as new at an idle point and carries its in-memory state into the next run:

• When:
  • its history passes 10,000 events (behind workflow.GetVersion); or
  • it receives the new continue-as-new signal (workflows.DeployContinueAsNewSignalName, no input).
• Only when idle: no deployment is running and the queue can't pop anything. The worker reports working while it waits on a locked queue with items, so idleness uses a new Worker.IsDeploying() plus Queue.CanPop() rather than GetState().
• What's carried over (deploy.Request.ContinuedState):
  • queued deployments, in pop order;
  • lock state;
  • check run cache entries, so queued deployments keep updating their existing check runs instead of creating duplicates.
• Before continuing: revision and unlock signals that arrived but weren't processed are handled, so they're carried over too.
• Metrics: the workflow interceptor counts continue-as-new separately instead of as a failure, plus a continue_as_new counter tagged with the reason.

Why there's a signal as well as the history check

GetVersion returns DefaultVersion when it's first called during replay, and keeps that for the rest of the execution (sdk-go v1.16.0 internal_event_handlers.go, GetVersion). A workflow started before this change replays its old timeouts first, so the history check never applies to it. The signal moves those workflows over once. Their histories contain no such signal, so listening for it doesn't affect replay. New runs, including those created by the signal, use the history check from then on.

Replay safety

On histories recorded before this change, nothing new emits a command:

• Registering the new signal channel and evaluating Idle(), HasPending() and the history length are pure.
• GetVersion returns the default during replay.
• Restoring state is a no-op when ContinuedState is nil.

The post-deploy PR revision child workflow is started with PARENT_CLOSE_POLICY_ABANDON, so it survives the parent continuing as new. The Terraform child workflow only runs while a deploy is in progress, which the idle check excludes.

Testing

New tests:

• Runner:

  • continues as new on a request while idle;
  • on a request while busy, waits for the next idle timeout;
  • never continues while never idle;
  • continues on history over the limit while idle;
  • doesn't continue over the limit on a workflow started before the change (GetVersion mocked to default);
  • doesn't continue under the limit;
  • doesn't continue over the limit while deploying.

  Each case decodes the continue-as-new input and checks the carried state and the original repo and root.

• queueContinuer.Snapshot drains a pending revision signal into the queue and applies a pending unlock.

• restoreQueue restores order and check run IDs, and is a no-op without state.

• Worker.IsDeploying is false for a worker that hasn't deployed, since InProgressStatus is the zero value.

Rollback

After a workflow has continued as new, code without this change would ignore ContinuedState on the next start and drop that queue. Fix forward rather than reverting once any workflow has continued.