Add experimental delivery ratio metrics for DANDI assets

CHANGELOG.md

@@ -4,6 +4,8 @@
 
 ### 🚀 Enhancement
 
+- Added an experimental, DANDI-only delivery ratio feature. The delivery ratio is the total bytes delivered over the asset's true DANDI size, signaling streaming versus download intensity. A per-asset `delivery_ratio` column is added to `by_asset.tsv`. The asset-weighted percentiles `delivery_ratio_p10`, `delivery_ratio_p25`, `delivery_ratio_p50`, `delivery_ratio_p75`, `delivery_ratio_p90` plus the volume-weighted `delivery_ratio_weighted` are injected into per-Dandiset `totals.json`, into archive `archive_totals.json`, and into a new archive `delivery_ratio.tsv`, via DANDI wrappers around the upstream summary and totals methods. Asset sizes are fetched live from the DANDI API on each run. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
+- Added the `dandis3logextraction update totals` command, with `--mode archive` for archive-wide totals, wrapping the upstream totals generation to also emit the delivery ratio fields. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
 - Added `download` to `_dandi_extraction.awk` so extraction writes `download.txt` alongside the other per-request outputs. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Added `--cache-directory` to `dandis3logextraction extract` so remote extraction can use a custom cache directory. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Added `--inventory` to `dandis3logextraction extract --mode remote` so extraction can use a local S3 Inventory directory. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
@@ -13,6 +15,10 @@
 - Refactored `generate_dandiset_totals` to derive the summary directory from `cache_directory`. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Renamed the `--directory` CLI flag to `--cache` in the update commands. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 
+### 📝 Documentation
+
+- Documented the experimental delivery ratio metric in the `README.md`. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
+
 ### 🔩 Dependency Updates
 
 - Updated compatibility for the latest `s3-log-extraction` release by pinning the lower bound to `>=1.9.2` and adapting extractor tests and summary columns. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))

README.md

@@ -71,3 +71,21 @@ dandis3logextraction update totals
 dandis3logextraction update summaries --mode archive
 dandis3logextraction update totals --mode archive
 ```
+
+
+
+## Delivery ratio (experimental)
+
+The delivery ratio is an experimental, DANDI-only signal of streaming versus download intensity.
+
+For each asset it is the total bytes delivered across all logged GET requests divided by the asset's true size in bytes. The true size comes from the DANDI API rather than the logs, and is fetched with a live lookup on each run. A ratio near 1 means access is download dominated. A ratio much greater than 1 means access is streaming dominated, since the asset's size worth of bytes has been delivered many times over across requests.
+
+The metric appears in three places:
+
+- A per-asset `delivery_ratio` column in each Dandiset's `by_asset.tsv`.
+- Per-Dandiset percentiles in `totals.json`, and archive wide percentiles in `archive_totals.json` and `archive/delivery_ratio.tsv`. The reported fields are `delivery_ratio_p10`, `delivery_ratio_p25`, `delivery_ratio_p50`, `delivery_ratio_p75`, `delivery_ratio_p90`, and `delivery_ratio_weighted`.
+
+Percentiles are reported at the Dandiset and archive level, rather than a single average, because per-asset delivery ratios are highly skewed. Within one Dandiset some assets are downloaded close to once while others are streamed many times over, so a mean would be dominated by a few heavily streamed assets and hide that spread. The five percentiles describe the shape of the distribution compactly, and the exact per-asset values are still available in `by_asset.tsv` for anyone who needs them.
+
+The percentiles are asset weighted, where each asset contributes one ratio. The `delivery_ratio_weighted` field is volume weighted, computed as the total bytes delivered over the total asset size. The gap between the weighted value and the median is a deliberate heterogeneity signal, so both are reported. Assets with a missing or zero size are excluded from the computation. A Dandiset with no usable asset reports empty values for all six fields.
+

pyproject.toml

@@ -15,7 +15,7 @@ allow-direct-references = true
 
 [project]
 name = "dandi-s3-log-extraction"
-version="1.0.2"
+version="1.0.3"
 authors = [
   { name="Cody Baker", email="cody.c.baker.phd@gmail.com" },
 ]
@@ -37,6 +37,7 @@ license = {file = "LICENSE.txt"}
 requires-python = ">=3.14"
 dependencies = [
     "beartype",
+    "numpy",
     "pandas",
     "tqdm",
     "PyYAML",

src/dandi_s3_log_extraction/_command_line_interface/_cli.py

@@ -8,7 +8,12 @@
 import s3_log_extraction
 
 from ..extractors import DandiRemoteS3LogAccessExtractor
-from ..summarize import generate_dandiset_summaries
+from ..summarize import (
+    generate_archive_summaries,
+    generate_archive_totals,
+    generate_dandiset_summaries,
+    generate_dandiset_totals,
+)
 
 
 # dandis3logextraction
@@ -231,7 +236,7 @@ def _update_summaries_cli(
     """Generate condensed summaries of activity."""
     match mode:
         case "archive":
-            s3_log_extraction.summarize.generate_archive_summaries(cache_directory=cache_directory)
+            generate_archive_summaries(cache_directory=cache_directory)
         case _:
             pick_as_list = pick.split(",") if pick is not None else None
             skip_as_list = skip.split(",") if skip is not None else None
@@ -244,3 +249,39 @@ def _update_summaries_cli(
                 unassociated=unassociated,
                 cache_directory=cache_directory,
             )
+
+
+# dandis3logextraction update totals
+@_update_cli.command(name="totals")
+@rich_click.option(
+    "--mode",
+    help=(
+        "Generate grand totals of activity across the extracted data. "
+        "Mode 'archive' aggregates over all Dandiset totals. "
+        "By default, per-Dandiset totals are generated."
+    ),
+    required=False,
+    type=rich_click.Choice(choices=["dandi", "archive"]),
+    default=None,
+)
+@rich_click.option(
+    "--cache",
+    "cache_directory",
+    help=(
+        "Path to the folder containing all previously extracted S3 access logs (`cache_directory`). "
+        "If not provided, the default cache directory from the configuration will be used."
+    ),
+    required=False,
+    type=rich_click.Path(file_okay=False, dir_okay=True),
+    default=None,
+)
+def _update_totals_cli(
+    mode: typing.Literal["dandi", "archive"] | None = None,
+    cache_directory: str | None = None,
+) -> None:
+    """Generate grand totals of all extracted data, including experimental delivery ratio fields."""
+    match mode:
+        case "archive":
+            generate_archive_totals(cache_directory=cache_directory)
+        case _:
+            generate_dandiset_totals(cache_directory=cache_directory)

src/dandi_s3_log_extraction/summarize/__init__.py

@@ -1,5 +1,13 @@
-from ._generate_dandiset_summaries import generate_dandiset_summaries
+from ._generate_dandiset_summaries import (
+    generate_archive_summaries,
+    generate_archive_totals,
+    generate_dandiset_summaries,
+    generate_dandiset_totals,
+)
 
 __all__ = [
+    "generate_archive_summaries",
+    "generate_archive_totals",
     "generate_dandiset_summaries",
+    "generate_dandiset_totals",
 ]