Add experimental delivery ratio metrics for DANDI assets - #86
Closed
CodyCBakerPhD wants to merge 14 commits into
Closed
Add experimental delivery ratio metrics for DANDI assets#86CodyCBakerPhD wants to merge 14 commits into
CodyCBakerPhD wants to merge 14 commits into
Conversation
Computed a per-asset delivery ratio (total bytes delivered over the asset's true DANDI size) and emitted it as a new column in by_asset.tsv. Added a per-Dandiset delivery_ratio.tsv with the asset-weighted percentiles (p10, p25, p50, p75, p90) plus a volume-weighted ratio. Asset sizes are fetched from the DANDI API and cached locally in asset_sizes.json, reused across runs. Assets with missing or zero size are excluded from the percentile computation and the skipped count is logged. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
Reworked the experimental delivery ratio feature so the per-Dandiset and archive percentiles live in the totals and archive outputs rather than a standalone per-Dandiset file. Added DANDI-specific wrappers generate_dandiset_totals, generate_archive_totals, and generate_archive_summaries that call the upstream s3_log_extraction methods and then inject the asset-weighted delivery ratio percentiles plus the volume-weighted ratio, sourced from each by_asset.tsv delivery_ratio column. Per-Dandiset fields are written into totals.json, archive-wide fields into archive_totals.json, and an archive delivery_ratio.tsv. Added the update totals CLI command and routed update summaries --mode archive through the DANDI wrapper. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
Removed the asset_sizes.json on-disk cache so asset sizes are looked up live from the DANDI API on every run; a persistent cache can be added later. Also reworded the README delivery ratio description to avoid implying the same byte ranges are re-served, since the logs do not distinguish ranges. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
CodyCBakerPhD
commented
Jun 22, 2026
Added a note to the README clarifying that per-asset delivery ratios are highly skewed, so the Dandiset and archive levels report distribution percentiles rather than a single average, while the per-asset values remain in by_asset.tsv. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
CodyCBakerPhD
commented
Jun 22, 2026
Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
…olumn The archive delivery_ratio.tsv now reports the five percentiles in a single "delivery_ratio(p10,p25,p50,p75,p90)" column with a comma-separated value rather than five separate columns, alongside the delivery_ratio_weighted scalar. The JSON totals keep individual keys. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
Pulled 1/s_a out of the per-asset equation and forced the summation indices below the sigma with \limits so they render aligned. Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
CodyCBakerPhD
commented
Jun 22, 2026
Co-Authored-By: Claude Code / Claude Opus 4.8 <noreply@anthropic.com>
CodyCBakerPhD
marked this pull request as draft
July 13, 2026 18:43
Collaborator
Author
|
Better researched and more conclusive strategy outlined on dandi/s3-log-extraction#284 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces an experimental delivery ratio feature that measures streaming versus download intensity for DANDI assets. The delivery ratio is calculated as total bytes delivered divided by the asset's true DANDI metadata size, providing insights into access patterns across the archive.
Key Changes
Asset Size Resolution: Added
_load_asset_sizes()function that fetches and caches DANDI asset sizes from the API, with persistent caching inasset_sizes.jsonto avoid repeated API calls at archive scale.Per-Asset Delivery Ratios: Extended
_summarize_dandiset_by_asset()to compute and output adelivery_ratiocolumn inby_asset.tsvfor each asset where size information is available.Percentile Aggregation: Implemented
_compute_delivery_ratio_percentiles()to calculate asset-weighted percentiles (p10, p25, p50, p75, p90) using linear interpolation, and_compute_delivery_ratio_fields()to compute both percentile-based and volume-weighted delivery ratios.Per-Dandiset Totals: Added
generate_dandiset_totals()function that augments per-Dandiset totals with delivery ratio fields derived from each Dandiset'sby_asset.tsv.Archive-Level Aggregation: Added
generate_archive_summaries()andgenerate_archive_totals()functions that pool delivery ratios across all Dandisets and output archive-wide delivery ratio metrics inarchive/delivery_ratio.tsvandarchive_totals.json.CLI Integration: Added new
update totalscommand with--modeoption to generate either per-Dandiset or archive-wide totals, with delivery ratio fields automatically included.Comprehensive Testing: Added
test_delivery_ratio.pywith 11 test cases covering percentile computation, asset filtering, field calculation, and archive pooling logic.Implementation Details
Noneto prevent re-fetching.delivery_ratiocolumn usesNaNfor assets with unresolved sizes, which are excluded from percentile calculations.NaNvalues tonullfor valid JSON output.https://claude.ai/code/session_012EXym6jB7G8NCt6vFMC8Gr