[improve][client] Make `Consumer.resume()` eagerly top up the broker's permit window

[PavelZeger]

Search before reporting

• I searched in the issues and found nothing similar.

Motivation

I was adding pause/resume to the Go client (apache/pulsar-client-go#1507), and while doing that I took a closer look at how Java's Consumer.resume() tells the broker to start sending messages again. I found a small thing worth improving, and it's the same shape as what I fixed on the Go side. When you resume, Java calls increaseAvailablePermits(cnx(), 0):

public void resume() {
    if (paused) {
        paused = false;
        increaseAvailablePermits(cnx(), 0);
    }
}

But that method only actually sends permits to the broker once the owed count reaches half the receiver queue:

while (available >= getCurrentReceiverQueueSize() / 2 && !paused) {
    ...
    sendFlowPermitsToBroker(currentCnx, available);
    ...
}

That "wait until half the queue" rule is good for normal running - you don't want to send a flow command for every single message. The thing is, it's also applied on resume. So if you resume at a moment when fewer than half a queue of permits are owed, resume sends nothing right then, and the broker's window only fills back up later as you keep consuming.

One correction to how I first described this: it is not a stuck/stall bug. As long as the permit invariant holds - brokerInFlight + queuedAtClient + availablePermits == currentReceiverQueueSize - the consumer always recovers on its own, and in Java the invariant does hold (duplicates, skipped batch entries, and intermediate chunks all give their permit back). I couldn't reproduce a stuck consumer, so I'm dropping that claim. What's left is a nice-to-have: let resume top the broker's window straight back up to the full queue size instead of waiting for the threshold. Thanks @lhotari for pointing out the wrong framing in my first version.

Solution

Give resume its own little flush that skips the half-queue rule and just sends whatever is owed. Because it only ever sends what's owed, it can never send too much:

@Override
public void resume() {
    if (paused) {
        paused = false;
        flushAvailablePermitsToBroker(cnx());
    }
}

private void flushAvailablePermitsToBroker(ClientCnx currentCnx) {
    int available = AVAILABLE_PERMITS_UPDATER.get(this);
    while (available > 0 && !paused) {
        if (AVAILABLE_PERMITS_UPDATER.compareAndSet(this, available, 0)) {
            sendFlowPermitsToBroker(currentCnx, available);
            break;
        } else {
            available = AVAILABLE_PERMITS_UPDATER.get(this);
        }
    }
}

A few notes:

• It uses the same compare-and-swap the existing code already uses, so there's no race and it won't send twice.
• The available > 0 check means it sends nothing when nothing is owed - including when an auto-scale-down has left availablePermits at zero or below.
• If the consumer happens to be disconnected, sendFlowPermitsToBroker does nothing and the reconnect path grants a fresh batch anyway, so that's safe.
• MultiTopicsConsumerImpl.resume() just calls resume on each child, so partitioned and multi-topic consumers are covered too.

This is the same fix I already used in the Go client (apache/pulsar-client-go#1507).

On the auto-scaled receiver queue (autoScaledReceiverQueueSizeEnabled(true)): I don't think it needs any special handling. The queue only grows and shrinks on the consume path (expectMoreIncomingMessages / reduceCurrentReceiverQueueSize), never through resume(). The flush sends exactly the owed permits, which by the invariant is always at most the current queue size, so it can't over-grant no matter what size scaling has it at. And the scaled-down case (where permits can go to zero or negative) is handled by the available > 0 guard.

On tests: the current testPauseAndResume drains the whole queue, so the owed count ends up back at a full queue size and the threshold path already flushes - which is exactly why the difference never shows today. A proper test needs to leave fewer permits owed than half the queue while the broker is at zero, then resume and check that a flow command actually goes out with exactly the owed count. The cleanest way is a ConsumerImplTest unit test with a mocked connection, plus an autoScaledReceiverQueueSizeEnabled(true) variant (scaled up: flush equals what's owed, nothing extra; scaled down with availablePermits <= 0: nothing sent).

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

But it's the wrong thing to do on resume. If, right when you resume, the consumer owes fewer permits than that and the broker has no permits left, then resume sends nothing.

I don't think that it's completely wrong and I don't see directly why the existing behavior would be causing the consumer to get stuck. However I do agree that it would make sense to send flow permits for all available permits after resuming the consumer.

When pause is called, the available permits should eventually get in the state where the consumer has available permits >= receiver queue size / 2. There must be other underlying bugs if the current logic doesn't resume the consumer.

When does this happen? Mostly when not every delivered message bumps the permit count by one (for example, duplicate or chunk fragments that get thrown away). In that case the owed count can end up below the threshold while the broker has already sent everything it was allowed to.

It seems that there are existing bugs if this happens. The permits handling should be only about what has been sent or received on the wire. If the broker sends duplicates, it shouldn't matter. Permit leaks would also impact other use cases, not just pause/resume. That's why finding and fixing the root cause is important.

In permit accounting the invariant is brokerInFlight + queuedAtClient + availablePermits == receiverQueueSize. A consumer is only truly stuck when the broker has 0 permits, the client queue is empty, and available permits < receiverQueueSize / 2. If the invariant holds, broker-idle + empty-queue forces available == receiverQueueSize, so resume would flush. A real stall therefore requires that this invariant is broken.

[PavelZeger]

You're right @lhotari , and I'll fix my framing.

The examples I gave don't actually leak in Java - duplicates, skipped batch entries, and intermediate
chunks all return their permit. So with the invariant holding, today's resume() would already flush. And I can't reproduce an actual stuck consumer, so I'll drop that claim.

I also can't really justify it on over-granting: Java's resume() already calls increaseAvailablePermits(cnx, 0), which sends only what's owed, not a full batch - so it can't over-grant today either. The only real difference my change can make is when owed < receiverQueueSize/2 on resume: current code waits, mine flushes right away. Without a leak that's not a stall, just a small thing - it tops the broker's window back to full the moment you resume instead of letting it refill as messages get consumed.

So I'm happy either way:

• if you think it's not worth it, I'll close this;
• if you think the eager top-up on resume earns its lines, I'll keep it minimal - the small flushAvailablePermitsToBroker change plus a ConsumerImplTest unit test, framed as "resume tops up the window" rather than a bug fix.

WDYT?

[lhotari]

• if you think the eager top-up on resume earns its lines, I'll keep it minimal - the small flushAvailablePermitsToBroker change plus a ConsumerImplTest unit test, framed as "resume tops up the window" rather than a bug fix.

I think that this can be justified. It would be necessary to ensure that the solution is also correct for the "autoscaled receiver queue" feature .autoScaledReceiverQueueSizeEnabled(true). I'm not sure if that requires any special handling.

[lhotari]

You're right @lhotari , and I'll fix my framing.

Please update the issue title and description

[PavelZeger]

You're right @lhotari , and I'll fix my framing.

Please update the issue title and description

Thanks for the review! I've changed the title and the description.