New architecture cannot resolve a pre-existing stale lock when the affected subscription remains uninitialized

[together-wang]

What did you do?

We ran a TiDB Cloud staging workload and performed a rolling upgrade of the upstream TiDB cluster. During the restart of one TiDB server, a pessimistic Async Commit transaction became undetermined and left stale locks before the TiCDC changefeed was created.

Afterward, we created a new-architecture TiCDC changefeed. One table subscription remained uninitialized, and both resolved-ts lag and checkpoint lag kept increasing.

Clinic evidence for the incident window (requires PingCAP internal access):

https://staging-clinic.pingcap.com/portal/#/orgs/10449/clusters/10221468604117851326?from=1786357800&to=1786375800

Timeline

1. TiDB received SIGTERM during transaction commit:

[2026/08/10 10:47:47.873 +00:00]
[INFO] [signal_posix.go:54]
["got signal to exit"]
[signal=terminated]

2. The transaction result became undetermined:

[2026/08/10 10:47:47.903 +00:00]
[ERROR] [2pc.go:1544]
["Async commit/1PC result undetermined"]
[error="context canceled"]
[rpcErr="context canceled"]
[txnStartTS=468283259055244183]

The same connection confirmed that this was a pessimistic transaction:

[2026/08/10 10:47:47.904 +00:00]
[WARN] [session.go:948]
["can not retry txn"]
[error="[global:2]execution result undetermined"]
[IsPessimistic=true]
[tidb_disable_txn_auto_retry=true]

3. The same transaction continued to block TiKV resolved-ts across multiple Regions:

[INFO] [endpoint.rs:612]
["the max gap of leader resolved-ts is large"]
[last_resolve_attempt="{ success=false, ts=468283259055244183, reason=lock, key=Some(?) }"]
[min_lock="Some((TimeStamp(468283259055244183), TxnLocks { lock_count: 1, sample_lock: Some(?) }))"]
[region_id=402657]

The same startTS was observed in Regions:

292561, 311203, 328639, 343251, 353327, 402657, 412401, 412413

4. After TiCDC started, one table subscription remained uninitialized:

[2026/08/10 13:45:32.228 +00:00]
[INFO] [event_store.go:1259]
["subscription lag snapshot"]
[initializedCount=5]
[uninitializedCount=1]
[uninitializedSubscriptions="[{\"subID\":8,\"tableSpan\":\"tableID: 11356, ...\"}]"]

5. We manually resolved only one Region using the exact stale transaction threshold:

cdc cli unsafe resolve-lock \
  --keyspace=default \
  --region=292561 \
  --ts=468283259055244184

TiCDC resolved it as an Async Commit transaction:

[2026/08/10 15:19:32.654 +00:00]
[INFO] [lock_resolver.go:1017]
["resolve async commit"]
[startTS=468283259055244183]
[commitTS=468283259055245754]

The request completed successfully in about 67 ms:

[2026/08/10 15:19:32.668 +00:00]
[INFO] [middleware.go:90]
["cdc open api request"]
[status=200]
[method=POST]
[path=/api/v2/unsafe/resolve_lock]
[duration=67.369132ms]

Immediately afterward, resolved-ts advanced and the subscription became working:

[2026/08/10 15:19:33.132 +00:00]
[WARN] [region_event_handler.go:404]
["resolved ts advance step is too large"]
[subID=8]
[tableID=11356]
[decreaseLag(s)=5656]

[2026/08/10 15:19:33.349 +00:00]
[INFO] [basic_dispatcher.go:364]
["update dispatcher status to working"]
[table="tableID: 11356, ..."]

After this single manual operation:

• the exact startTS disappeared from all affected Regions;
• TiKV resolved-ts lag dropped from about 16,255,800 ms to 50 ms;
• the table subscription changed from uninitialized to working;
• the changefeed remained in normal state.

What did you expect to see?

TiCDC should automatically resolve a pre-existing stale transaction lock even when the lock existed before changefeed creation.

At minimum, an uninitialized subscription should not remain permanently blocked because automatic stale-lock resolution requires that same subscription to be initialized first.

What did you see instead?

The subscription remained uninitialized and lag increased continuously until a manual cdc cli unsafe resolve-lock operation was performed.

Automatic retry behavior observed

There are three different mechanisms here, and they should not be conflated:

• The original TiDB transaction did not retry. TiDB logged can not retry txn with tidb_disable_txn_auto_retry=true after the Async Commit/1PC result became undetermined.
• TiKV repeatedly attempted to advance Region resolved-ts. For example, Region 311203 reported the same last_resolve_attempt={ success=false, ts=468283259055244183, reason=lock } at 10:48:18.417, 10:48:38.425, and subsequent approximately 20-second intervals. This shows that TiKV kept detecting the blocking lock; it does not show that TiCDC successfully ran stale-lock resolution.
• TiCDC's automatic checker is scheduled every two seconds, but it did not make an effective resolve-lock attempt for this subscription. The affected subscription 8 / table 11356 remained uninitialized, and getResolvedTargetTs returns 0 when subSpan.initialized is false. Therefore no target TS is passed to resolveStaleLocks, and no Region resolve-lock task can be scheduled from this subscription.

In other words, the automatic check loop can continue ticking while the actual stale-lock resolution path is short-circuited by the initialization guard. The manual unsafe API bypassed this gate and resolved the same Async Commit transaction immediately.

The checker interval and initialization guard are visible here:

ticdc/logservice/logpuller/subscription_client.go

Lines 53 to 60 in 95374f0

	// don't need to force reload region anymore.
	regionScheduleReload = false
	loadRegionRetryInterval time.Duration = 100 * time.Millisecond
	resolveLockMinInterval time.Duration = 10 * time.Second
	resolveLockTickInterval time.Duration = 2 * time.Second
	resolveLockFence time.Duration = 4 * time.Second
	)

The running version contains two initialization guards in the automatic stale-lock resolution path:

if !subSpan.initialized.Load() || time.Since(resolvedTsUpdated) < resolveLockFence {
    return 0
}

ticdc/logservice/logpuller/subscription_client.go

Lines 887 to 904 in 95374f0

	func (s *subscriptionClient) runResolveLockChecker(ctx context.Context) error {
	resolveLockTicker := time.NewTicker(resolveLockTickInterval)
	defer resolveLockTicker.Stop()
	maxCacheSize := 1024
	subSpanAndTsCache := make([]subscriptionAndTargetTs, 0, maxCacheSize)
	// getResolvedTargetTs returns the targetTs to resolve stale locks. 0 means no need to resolve.
	getResolvedTargetTs := func(subSpan *subscribedSpan, currentTime time.Time) uint64 {
	resolvedTsUpdated := time.Unix(subSpan.resolvedTsUpdated.Load(), 0)
	if !subSpan.initialized.Load() || time.Since(resolvedTsUpdated) < resolveLockFence {
	return 0
	}
	resolvedTs := subSpan.resolvedTs.Load()
	resolvedTime := oracle.GetTimeFromTS(resolvedTs)
	if currentTime.Sub(resolvedTime) < resolveLockFence {
	return 0
	}
	return oracle.GoTimeToTS(resolvedTime.Add(resolveLockFence))
	}

The later scheduling path also requires the Region state to be initialized:

if state.ResolvedTs.Load() < targetTs && state.Initialized.Load() {
    // schedule resolve-lock
}

ticdc/logservice/logpuller/subscription_client.go

Lines 1051 to 1063 in 95374f0

	rt.tryResolveLock = func(regionID uint64, state *regionlock.LockedRangeState) {
	targetTs := rt.staleLocksTargetTs.Load()
	if state.ResolvedTs.Load() < targetTs && state.Initialized.Load() {
	select {
	case <-s.ctx.Done():
	case s.resolveLockTaskCh <- resolveLockTask{
	keyspaceID: span.KeyspaceID,
	regionID: regionID,
	targetTs: targetTs,
	state: state,
	create: time.Now(),
	}:
	// it is ok to ignore resolve lock task when the channel is full

This appears to create a self-blocking cycle:

pre-existing stale lock
  -> subscription cannot initialize
  -> automatic resolver skips the uninitialized subscription
  -> stale lock is never resolved
  -> subscription remains uninitialized

Difference from #5418

This does not appear to be a direct reproduction of #5418:

• no commit_ts_expired was found for the target transaction;
• no matching commit_ts < min_commit_ts error was found for this startTS;
• manual resolve-lock successfully committed the Async Commit transaction.

The suspected issue here is specifically that the automatic resolver skips the uninitialized subscription that is blocked by the stale lock.

Versions of the cluster

Upstream TiDB cluster version:

Release Version: v8.5.8
Git Commit Hash: f5fa13c9e127e734a94367cca1d0a3dfb4967710

Upstream TiKV version:

Release Version: v8.5.8
Git Commit Hash: 6fb55a0abb77ff8a1b2b531974a01553daafa605

TiCDC version (verified by entering both running TiCDC Pods and executing /cdc version):

Image: gcr.io/pingcap-public/dbaas/ticdc:v8.5.6-release.1
Release Version: v8.5.6
Git Commit Hash: 95374f068094a71d1fb0d68b8cc141645d008938
UTC Build Time: 2026-01-31 09:13:45
Go Version: go1.25.5 linux/amd64

Both TiCDC Pods were running the same image and commit.

[ti-chi-bot]

Welcome @together-wang! It looks like this is your first issue to pingcap/ticdc 🎉