Repository navigation
Bump Temporal Go SDK to v1.16.0 and add a workflow replay test - #793
Merged
Merged
Conversation
The deploy and PR workflows run for a long time and never continue as new, so their histories grow without bound. Fixing that needs workflow.GetInfo(ctx).GetCurrentHistoryLength(), which the SDK added in v1.16.0. This bumps go.temporal.io/sdk v1.15.0 -> v1.16.0 (and go.temporal.io/api v1.8.0 -> v1.11.0, gogo/status v1.1.0 -> v1.1.1) with no workflow code changes. An SDK bump or workflow change that alters the commands a workflow emits breaks workflows that are already running. TestReplayHistories replays exported histories against the current code to catch that. Histories hold production workflow inputs, so the test reads them from the directory in ATLANTIS_REPLAY_HISTORY_DIR and skips when it isn't set. The test registers workflows the way the worker does. The Deploy workflow and its plugins are moved into deployWorkflow() so the worker and the test share one definition. Current Temporal CLI exports name enums like EVENT_TYPE_WORKFLOW_EXECUTION_STARTED and include fields newer than this API version, so the test rewrites enum names and ignores unknown fields, as the worker does when it receives history as protobuf.
gosec G703 flagged os.ReadFile on a path built from ATLANTIS_REPLAY_HISTORY_DIR. Opening the directory with os.OpenRoot confines every read to it, so a history name can't reach outside the directory.
emmanuelmaduwuba-lyft
added this pull request to stack #795
October 8, 2026 15:57
Contributor
Author
|
/ptal @yusyuan-lyft @jfernandoz |
yusyuan-lyft
approved these changes
Oct 8, 2026
emmanuelmaduwuba-lyft
added a commit
that referenced
this pull request
Oct 8, 2026
**Why** The Deploy workflow (one per repo and root) only exits when its queue is empty (`deploy/workflow.go`, `OnTimeout`). A queue that is locked with merged revisions waiting keeps the workflow alive indefinitely, and each idle hour adds a timer and a workflow task to its history, roughly 125 events a day. Two things go wrong as it grows: 1. **Past Temporal's hard history limit, the server terminates the workflow.** The queue and lock live only in workflow memory (`queue/worker.go`: "we don't persist lock state anywhere"), so queued deploys are dropped without any notice. 2. **A large history can stop a workflow from making progress.** After a worker restart, replaying it can exceed the SDK's 1-second deadlock detection, and every workflow task then fails. **What changes** The workflow continues as new at an idle point and carries its in-memory state into the next run: - **When:** - its history passes 10,000 events (behind `workflow.GetVersion`); or - it receives the new `continue-as-new` signal (`workflows.DeployContinueAsNewSignalName`, no input). - **Only when idle:** no deployment is running and the queue can't pop anything. The worker reports `working` while it waits on a locked queue with items, so idleness uses a new `Worker.IsDeploying()` plus `Queue.CanPop()` rather than `GetState()`. - **What's carried over** (`deploy.Request.ContinuedState`): - queued deployments, in pop order; - lock state; - check run cache entries, so queued deployments keep updating their existing check runs instead of creating duplicates. - **Before continuing:** revision and unlock signals that arrived but weren't processed are handled, so they're carried over too. - **Metrics:** the workflow interceptor counts continue-as-new separately instead of as a failure, plus a `continue_as_new` counter tagged with the reason. **Why there's a signal as well as the history check** `GetVersion` returns `DefaultVersion` when it's first called during replay, and keeps that for the rest of the execution (sdk-go v1.16.0 `internal_event_handlers.go`, `GetVersion`). A workflow started before this change replays its old timeouts first, so the history check never applies to it. The signal moves those workflows over once. Their histories contain no such signal, so listening for it doesn't affect replay. New runs, including those created by the signal, use the history check from then on. **Replay safety** On histories recorded before this change, nothing new emits a command: - Registering the new signal channel and evaluating `Idle()`, `HasPending()` and the history length are pure. - `GetVersion` returns the default during replay. - Restoring state is a no-op when `ContinuedState` is nil. The post-deploy PR revision child workflow is started with `PARENT_CLOSE_POLICY_ABANDON`, so it survives the parent continuing as new. The Terraform child workflow only runs while a deploy is in progress, which the idle check excludes. **Testing** New tests: - Runner: - continues as new on a request while idle; - on a request while busy, waits for the next idle timeout; - never continues while never idle; - continues on history over the limit while idle; - doesn't continue over the limit on a workflow started before the change (`GetVersion` mocked to default); - doesn't continue under the limit; - doesn't continue over the limit while deploying. Each case decodes the continue-as-new input and checks the carried state and the original repo and root. - `queueContinuer.Snapshot` drains a pending revision signal into the queue and applies a pending unlock. - `restoreQueue` restores order and check run IDs, and is a no-op without state. - `Worker.IsDeploying` is false for a worker that hasn't deployed, since `InProgressStatus` is the zero value. `go test ./...` passes except two legacy e2e tests that need a released `conftest` on `PATH`, which fail the same way on `main`. Before merging, run `TestReplayHistories` from #793 against exported production Deploy histories, including a long-running one. **Rollout** 1. Deploy to staging, then production. 2. New deploy workflows continue as new automatically once they pass 10,000 events. 3. Send `continue-as-new` once to each existing long-running deploy workflow, e.g. `temporal workflow signal --workflow-id '<repo>||<root>' --name continue-as-new`. Then check the new run's start input carries the expected queue, and that queued check runs are updated, not duplicated. **Rollback** After a workflow has continued as new, code without this change would ignore `ContinuedState` on the next start and drop that queue. Fix forward rather than reverting once any workflow has continued.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Deploy workflow (one per repo and root) and the PR workflow run for a long time and never call
ContinueAsNew, so their histories grow without bound. A large history can leave a workflow unable to make progress after a worker restart, because replaying it exceeds the SDK's 1-second deadlock detection. Past Temporal's hard history limit, the server terminates the workflow and any queued deploys are lost. The fix, in a follow-up PR, continues these workflows as new once their history passes a threshold. That needsworkflow.GetInfo(ctx).GetCurrentHistoryLength(), which the SDK added in v1.16.0.What changes
go.temporal.io/sdkv1.15.0 → v1.16.0,go.temporal.io/apiv1.8.0 → v1.11.0,github.com/gogo/statusv1.1.0 → v1.1.1. No workflow code changes.TestReplayHistoriesreplays exported workflow histories against the current code, registered the way the worker registers it. Histories hold production workflow inputs, so they're read fromATLANTIS_REPLAY_HISTORY_DIRand never checked in. The test skips when the variable isn't set.deployWorkflow(), so the worker and the replay test share one definition.EVENT_TYPE_WORKFLOW_EXECUTION_STARTEDand include fields newer than API v1.11. The test rewrites the enum names and ignores unknown fields, which is what the worker does when it receives history as protobuf.v1.16.0 release notes worth a look
Failure. Check that nothing downstream parses the old error shape.Testing
go build ./...succeeds with the existingreplace google.golang.org/grpc => v1.45.0pin, andgo vet ./server/neptune/...is clean.go test ./...passes except two legacy e2e tests that need a releasedconftestbinary onPATH; they fail the same way onmain