Reader intermittently returns incomplete results when reading from earliest on a topic with S3-offloaded messages

[Goodkat]

Environment

• Pulsar version: 4.0.13
• Client: pulsar-client Python 3.10.0
• Topic: Persistent, partitioned (1 partition)
• Reader config: Starts from MessageId.earliest, no subscription

Topic configuration

Retention:

{
  "retentionTimeInMinutes": 5300000,
  "retentionSizeInMB": -1
}

TTL:

• Original: 2 weeks
• Workaround: 1 year (extending TTL beyond the age of the earliest message eliminates the issue)

Offload (S3):

{
  "managedLedgerOffloadThresholdInSeconds": 259200,
  "managedLedgerOffloadThresholdInBytes": 1073741824,
  "managedLedgerOffloadMaxThreads": 2,
  "managedLedgerOffloadReadThreads": 2,
  "managedLedgerOffloadPrefetchRounds": 1,
  "s3ManagedLedgerOffloadMaxBlockSizeInBytes": 67108864,
  "s3ManagedLedgerOffloadReadBufferSizeInBytes": 1048576
}

Problem

A reader that starts from MessageId.earliest and reads all messages until the end of the topic intermittently returns far fewer messages than expected. The topic contains ~24,747 messages (with 9 new messages arriving after ~12:00, bringing the total to 24,756). However, approximately 40% of runs return only a fraction of the messages, while the remaining 60% return the correct count.

Given the long retention period and the offload configuration, a significant portion of the messages are offloaded to S3. The incomplete runs seem to correlate with the reader failing to read some or all of the offloaded ledgers.

Reproduction data

We ran the reader 10 times consecutively. 4 out of 10 runs returned incomplete results:

Run	Time	Messages returned	Status
1	11:30	9,281	Incomplete
2	11:38	8,942	Incomplete
3	11:45	24,747	OK
4	11:50	24,747	OK
5	12:00	2,805	Incomplete
6	12:05	2,647	Incomplete
7	12:08	24,756	OK
8	12:12	24,756	OK
9	12:16	24,756	OK
10	12:18	24,756	OK

The "OK" runs after 12:00 return 24,756 (= 24,747 + 9 new messages), confirming that all messages are present and retention is not the issue. The incomplete runs return wildly varying counts (2,647 to 9,281), suggesting the reader silently stops or skips ledgers rather than producing an error.

Key finding: extending TTL eliminates the problem

After extending the TTL from 2 weeks to 1 year (so that the TTL boundary is well beyond the oldest message in the topic), we can no longer reproduce the issue. All reader runs return the full message count.

This strongly suggests that the TTL expiry background task interferes with the reader when it is scanning through messages that have aged past the TTL boundary.

Observations

• Not a retention issue: Successful runs return the full message count, confirming all messages are retained. The retention period (~3,680 days) is far longer than the age of any message.
• TTL should not delete retained messages: TTL only force-acknowledges unacknowledged messages. After acknowledgment, those messages fall under the retention policy. Since retention covers all messages, no data should be removed. However, something about the TTL expiry process appears to interfere with the reader's ability to traverse those ranges.
• The issue is a race condition between the TTL expiry task and the reader. We suspect the following sequence:
  1. A Reader starts from MessageId.earliest and begins scanning forward through ledgers (including offloaded ones on S3).
  2. The broker's scheduled TTL expiry task runs concurrently and force-acknowledges messages older than the TTL boundary, advancing the internal cursor.
  3. The Reader, which has no subscription and should be independent of cursor movements, is nonetheless affected — it appears to skip or lose access to ranges of messages that the expiry task has just processed.
  4. The result depends entirely on timing: if the Reader finishes scanning before the expiry task runs (or if TTL is extended so the task has nothing to expire), all messages are returned correctly.

Concerns

This behavior raises several concerns about the Reader's contract:

1. A Reader can start before the TTL boundary. A reader starting from MessageId.earliest is expected to see all retained messages, regardless of their acknowledgment state. TTL expiry should not affect what a reader can see, since readers operate outside the subscription model.

2. The broker can move its active cursor during the scan. If the TTL expiry task advances internal cursors while a reader is mid-scan, the reader may lose its position or skip ledgers. This breaks the expectation of a stable, sequential read.

3. The Reader receives no indication that a range was skipped. The reader does not throw an error, log a warning, or return any signal that messages were missed. It silently returns a partial result set, making this very difficult to detect in production.

4. Results depend on whether the scheduled expiry task runs before the Reader catches up. This makes the behavior non-deterministic and timing-dependent — identical reader code on the same topic can return 2,647 messages or 24,756 messages depending on when the TTL background task last ran.

Questions

1. Is this a known interaction between the TTL expiry task and readers? Should readers be immune to cursor movements caused by TTL expiry?
2. Is there a mechanism by which the TTL task's cursor advancement could cause a reader to skip ledgers, particularly offloaded ones on S3?
3. Should the reader signal an error or warning when it encounters a gap in the ledger chain, rather than silently returning fewer messages?
4. Are there any broker-side configurations (beyond extending TTL) that could prevent this race condition?
5. Are there broker logs we should collect during an incomplete run that would confirm the TTL task interfered with the reader's scan?

Any guidance would be appreciated. Thank you!

[lhotari]

This seems to be a bug.

• Client: pulsar-client Python 3.10.0

Can you reproduce this also with Python client 3.13.0?

[lhotari]

• The issue is a race condition between the TTL expiry task and the reader.

This seems like a correct observation. Most likely the Python client isn't involved.

[lhotari]

@Goodkat Thanks for the great issue report. I created this issue based on your description: #26504. The fix is in #26505.