add live publications load test - #92
Merged
Merged
Conversation
…bjective CCWG#15 sets a service objective of a 150 ms p90 for requests carrying up to 100 publication identifiers, and the README has been carrying a note that the deployment still needed a benchmark to verify it. This adds one, scoped to /publications because that endpoint is the drop-in replacement for DocumentMetadataAPI and shares no code path with the generic annotation routes. benchmarks/publications is a closed-loop async driver with a CLI. Closed-loop rather than an open-loop arrival rate: it answers the question the objective actually poses -- how many concurrent users fit inside the budget -- and cannot drive the service into an unbounded queue and then report the queue delay as service latency. Four measurement choices, each of which would otherwise flatter the result: - Each stage discards a warmup so no measured sample carries a TLS handshake. The warmup draws different identifiers than the measured stage, so it warms the connection pool without priming the backend cache for the very lookups under measurement. - Identifiers are drawn fresh by default. Elasticsearch caches hard enough that a replayed batch runs at under half the cold cost, so reusing identifiers would report the best case as if it were typical. A verified corpus is available but labels its run cache_primed, because verification is itself a query that warms the cache. - The hit ratio is reported. An all-not_found response is cheap to serve, so a run whose identifiers mostly missed could otherwise show a fast p90 for work the service never performed. - Percentiles are nearest-rank, and the fraction of requests strictly under the threshold is reported alongside. That fraction is the more literal reading of "90% of requests should take <150ms" and needs no interpolation convention to reproduce. Client-side and server-side latency are both recorded because they disagree and each answers a different question. A full batch returns ~115 kB of abstracts, so from outside the cluster transit alone adds 150-250 ms at p90; the report prints that difference so a client-side miss is never mistaken for a service regression. The exit code gates on the server-side figure by default, which is comparable from any vantage point. Measured against CI: the PMID fast path meets the objective with roughly half the budget spare (100 identifiers at p90 73 ms) and scales sublinearly with batch size. The binding constraint is concurrency, not batch size -- the knee sits between 8 and 12 concurrent full batches at ~45 rps, past which throughput stops rising while latency climbs, which is queueing rather than slower work. Mixed PMCID and DOI batches pass at p90 but carry a far heavier tail (p95 181 ms against 80 ms for PMID-only) from the per-identifier _msearch path; that is the workload to re-measure once the reindex makes those identifiers resolve, since today's cheap misses become real lookups. The harness is also wrapped as pytest. Its own percentile arithmetic, verdict boundaries, and the accounting that keeps a failed or malformed 200 from scoring as a fast request are covered offline, because a benchmark that is wrong about those reports confidence it has not earned. The live tests are opt-in behind RUN_PUBLICATIONS_LOAD_TEST, following the RUN_PUBMED_ES_INTEGRATION convention.
…a byte rate The first write-up reported the knee as sitting "between 8 and 12 concurrent full batches" and concluded that concurrency was the binding constraint. That conclusion does not survive the control: running the same concurrency ramp with a single-identifier batch scales linearly to 221 rps at c=32 with a flat 6 ms p90 and no knee anywhere. Nothing about 12 concurrent requests is hard for the service, so holding batch size at 100 had made concurrency look like the variable when payload size was. The ceiling is a response byte rate of roughly 5 MB/s per pod, and the payload is dominated by one field: abstracts are 75% of a 100-identifier response, titles 8%. That reframes the levers. CI runs replicaCount: 1 and the work is per-pod -- Sanic serializing ~119 kB of JSON, Caddy compressing it 2.9x under 8 workers -- so replicas should scale it close to linearly, but field selection would move it further than replicas for callers that do not need abstracts when rendering a list. Also records what the number does and does not mean. At 46 full batches per second, about 4,500 publication lookups per second on one pod, the ceiling sits well above expected UI traffic; the part that deserves attention is its shape, 89 ms to 306 ms to 720 ms with no graceful degradation and no load shedding. And the attribution is stated with its limit: processing_time_ms is measured before the response is written so a slow client cannot inflate it directly, but event-loop time draining large responses can still delay a worker's other requests, and separating CPU from egress needs pod metrics the benchmark does not collect.
…count The concurrency ramp answers "what is the ceiling" and answers it by holding N requests permanently in flight. That is not a model of N people: a closed-loop worker issues its next request the instant the previous returns, so it behaves like an infinitely impatient robot. A reader pausing 30 s between lookups offers about 0.03 rps against the ~11 rps a saturated worker offers -- nearly three orders of magnitude, which is exactly the gap that made the earlier "knee at c=12" reporting sound alarming when it described no user-facing limit at all. Adds a user-population driver alongside the closed-loop one, and the Little's Law translation (users = throughput x (latency + think time)) so the two can be read against each other. Three details separate it from N copies of one user: think times are exponentially distributed and arrivals staggered, so the population does not run in lockstep or open with a thundering herd; users share one Zipf-ranked catalogue, because overlapping reader interest is what a backend cache actually absorbs and independent per-user draws hide it; and the report distinguishes offered from achieved rate, so a population that was served without saturating the service is not mistaken for a ceiling. Measured against CI, a realistic population reaches ~90 rps at a 66 ms p90, about double the 46 rps cold-cache closed-loop ceiling. Both effects behind the gap are properties of a real population rather than measurement error: a skewed draw collapses repeated popular papers, so a 100-identifier request sends ~79 distinct ones, and the shared hot set stays cached. The cold-cache ramp is a conservative bound, not the expected operating point. In user terms the 46 rps figure supports roughly 1,400 readers at a 30 s think time, well above expected UI traffic. Two corrections to the harness fall out of running it. Near saturation one run shed ~1.5% of requests as HTTP 500s -- the two-second backend deadline firing -- and the capacity figure counted that stage anyway, because failures are excluded from the latency distribution and a shedding stage therefore posts a *better* p90 than one serving everything slowly. Capacity is now gated on a 99% success rate and any stage passing on latency while shedding is flagged inline. Second, at these rates a single load generator is itself a plausible bottleneck (~8 MB/s to decompress and parse), so the report uses in-handler server latency as the discriminator: a shortfall alongside a healthy server-side p90 and no errors is attributed to the generator rather than credited to the service as a ceiling it was never pushed to.
Every simulated user issues one request the moment it joins and only then settles into its think-time cadence, so a short run is dominated by those joining requests. Over a window of k think times the observed rate runs about 1 + 1/(2k) times the steady-state rate: 25% high at k=2, 5% at k=10. The sweep that exposed this offered 10 rps from 300 users and measured 12.4 -- an impossible result in steady state, and the arithmetic predicts 746 requests against the 743 observed. Left alone this made the report's headline population verdict unsafe in the one direction that matters: a genuinely saturated population reads as though it kept up. UserModel now exposes the run length in think times, and the report declines to render a saturation verdict below ten of them, saying instead how high the rate reads and how long to run for a trustworthy comparison. Latency is explicitly exempted -- each sample is an independent per-request measurement and stays valid on a short run, which is what keeps a quick run useful for the SLO question even when it cannot speak to throughput.
…urements The committed write-up quoted a knee at 12 concurrent requests and a ~5 MB/s per-pod byte-rate ceiling. Repeating the ramp three times does not reproduce either. Under smooth closed-loop load the objective holds to roughly 100 rps at 24 concurrent full batches and the p90 reaches the 150 ms line around 32; the run behind the original claim was about an order of magnitude worse at the same concurrency, with throughput plateauing rather than climbing. CI is a shared single-replica deployment measured from outside the cluster, so that run is an environment sample rather than a property of the service. The section now shows all three runs side by side with the single-identifier control, and says to repeat the ramp before quoting a ceiling. The population figures are re-measured over 300 s rather than 60 s so the joining transient no longer biases the rate. About 1,200 concurrent readers at a 30 s think time fit inside the objective at a 28 ms p90; 2,400 offer 80 rps, receive 53, and miss at 808 ms. Two findings come out of having both drivers. The population misses at 53 rps where the smooth ramp sustained ~100, which is burstiness rather than a contradiction: exponential think times give Poisson arrivals whose instantaneous rate clumps far above its mean, and those clumps queue behind the 8 workers, so the smooth ramp is the optimistic bound and the bursty figure the realistic one. And 1,200 readers post a better p90 than 100 do, because at a low request rate the shared hot set falls out of cache between requests -- a lightly loaded service does more cold lookups than a moderately loaded one.
Resolves DOI and PMCID to document _ids with a source-free _msearch, then fetches every document -- resolved and submitted PMIDs alike -- through one _mget on the exact-ID path. Off by default behind DOCUMENT_METADATA_TWO_PHASE_LOOKUP so the two strategies can be measured against each other on a real deployment. querymany_ids is a distinct client method rather than querymany(fields=[]), because an empty source filter still has Elasticsearch load _source and then filter it away. "_source": false skips loading it, which for this index is the difference between decompressing every matched abstract and touching none. Flagged rather than substituted because the trade runs both ways. It removes source loading from the search leg and collapses two source fetches into one batched request, but it serializes what the current path runs concurrently: the _mget cannot start until the _msearch returns. A PMID-heavy batch today hides the entire DOI lookup behind the PMID fetch and would lose that overlap; a DOI-heavy batch has little overlap to lose. Which wins is workload dependent. It also does not address the _msearch's dominant cost, and the docstring says so: a term query on pubmed.identifiers carries no routing key, so every entry executes on every shard however little it returns. Only the fetch phase gets cheaper. Removing the fan-out would mean making DOI and PMCID resolvable by _id. The tests cover where this strategy can silently go wrong rather than just the happy path. Resolving to a document _id and fetching by it discards the submitted identifier unless the mapping is carried through, which would key a result under a PMID the caller never asked about; several submitted identifiers can name one document, so the fetch must deduplicate while both keys still receive it; and one case asserts the flag changes only how the lookup is issued, never the result.
…search Deduplicating pubmed.identifiers is a storage change, not a latency one, and the numbers now say so rather than leaving it to be quoted either way. Built two indices from the same 29,983 real documents with the live mapping and analysis settings, differing only in whether the PMID is also present in identifiers, and force-merged both: the field roughly halves (43.5M values against 84.5M with the PMID re-added), which is 1.0 GB per copy and 3.0 GB across three copies, or 2.3% of the index. Latency is unchanged and measurably so -- identical _msearch took at 100 identifiers, and a _mget response 1% smaller. The two-phase lookup is rejected on measurement and stays off. Its premise was that fetching source on the search leg is expensive; it is not. A 100-identifier search that resolves every one takes 22 ms of Elasticsearch wall-clock and "_source": false removes about 1 ms. The 84% cut in that leg's response size is not a saving either, because the bytes reappear in the larger _mget plus its per-document envelope, so total bytes rise 3-8% depending on mix. Against no gain it adds a serial round trip and forfeits the concurrency that hides the whole DOI lookup behind the PMID fetch. Two earlier claims are corrected. The shard fan-out is not the dominant cost: per-identifier time falls from 0.40 ms at 10 identifiers to 0.22 ms at 100, so the sub-searches parallelise well across the 5 shards. And identifier resolution is not where the budget goes -- 22 ms for a fully-resolving 100-identifier batch against the ~114 kB of response body the load benchmark identifies as the real constraint, so effort belongs on payload size. Also records why the earlier mixed-identifier measurements could not have answered this. Through an SSH tunnel each Elasticsearch round trip costs 90-190 ms and drifts during a session, more than the entire effect being measured, so these figures come from the top-level took on the _msearch response rather than from wall-clock. Per-sub-request took values are not summed, since Elasticsearch runs msearch entries concurrently and their sum is work rather than latency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add paired publications lookup benchmark