Add experimental delivery ratio metrics for DANDI assets

CHANGELOG.md

@@ -4,6 +4,8 @@
 
 ### 🚀 Enhancement
 
+- Added an experimental, DANDI-only delivery ratio feature. The delivery ratio is the total bytes delivered over the asset's true DANDI size, signaling streaming versus download intensity. A per-asset `delivery_ratio` column is added to `by_asset.tsv`. The asset-weighted percentiles `delivery_ratio_p10`, `delivery_ratio_p25`, `delivery_ratio_p50`, `delivery_ratio_p75`, `delivery_ratio_p90` plus the volume-weighted `delivery_ratio_weighted` are injected into per-Dandiset `totals.json`, into archive `archive_totals.json`, and into a new archive `delivery_ratio.tsv`, via DANDI wrappers around the upstream summary and totals methods. Asset sizes are fetched live from the DANDI API on each run. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
+- Added the `dandis3logextraction update totals` command, with `--mode archive` for archive-wide totals, wrapping the upstream totals generation to also emit the delivery ratio fields. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
 - Added `download` to `_dandi_extraction.awk` so extraction writes `download.txt` alongside the other per-request outputs. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Added `--cache-directory` to `dandis3logextraction extract` so remote extraction can use a custom cache directory. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Added `--inventory` to `dandis3logextraction extract --mode remote` so extraction can use a local S3 Inventory directory. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
@@ -13,6 +15,10 @@
 - Refactored `generate_dandiset_totals` to derive the summary directory from `cache_directory`. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 - Renamed the `--directory` CLI flag to `--cache` in the update commands. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))
 
+### 📝 Documentation
+
+- Documented the experimental delivery ratio metric in the `README.md`. ([#86](https://github.com/dandi/dandi-s3-log-extraction/pull/86))
+
 ### 🔩 Dependency Updates
 
 - Updated compatibility for the latest `s3-log-extraction` release by pinning the lower bound to `>=1.9.2` and adapting extractor tests and summary columns. ([#68](https://github.com/dandi/dandi-s3-log-extraction/pull/68))

README.md

@@ -71,3 +71,33 @@ dandis3logextraction update totals
 dandis3logextraction update summaries --mode archive
 dandis3logextraction update totals --mode archive
 ```
+
+
+
+## Delivery ratio (experimental)
+
+The delivery ratio is an experimental, DANDI-only signal of streaming versus download intensity.
+
+For each asset it is the total bytes delivered across all logged GET requests divided by the asset's true size in bytes:
+
+$$r_a = \frac{1}{s_a} \sum\limits_{n \in G_a} b_n$$
+
+where:
+- $G_a$ is the set of logged GET requests for asset $a$
+- $b_n$ is the bytes delivered in request $n$
+- $s_a$ is the asset's true size in bytes.
+
+A ratio near 1 means access is download dominated. A ratio much greater than 1 means access is streaming dominated, since the asset's size worth of bytes has been delivered many times over across requests.
+
+The metric appears in three places:
+
+- A per-asset `delivery_ratio` column in each Dandiset's `by_asset.tsv`.
+- Per-Dandiset percentiles in `totals.json`, and archive wide percentiles in `archive_totals.json` and `archive/delivery_ratio.tsv`. The reported fields are `delivery_ratio_p10`, `delivery_ratio_p25`, `delivery_ratio_p50`, `delivery_ratio_p75`, `delivery_ratio_p90`, and `delivery_ratio_weighted`.
+
+Percentiles are reported at the Dandiset and archive level, rather than a single average, because per-asset delivery ratios are highly skewed. Within one Dandiset some assets are downloaded close to once while others are streamed many times over, so a mean would be dominated by a few heavily streamed assets and hide that spread. The five percentiles describe the shape of the distribution compactly, and the exact per-asset values are still available in `by_asset.tsv` for anyone who needs them.
+
+The percentiles are asset weighted, where each asset contributes one ratio $r_a$ regardless of its size. The `delivery_ratio_weighted` field is instead volume weighted, computed as the total bytes delivered over the total asset size across the usable assets $A$:
+
+$$r_{\text{vol}} = \frac{\sum\limits_{a \in A} \sum\limits_{n \in G_a} b_n}{\sum\limits_{a \in A} s_a}$$
+
+The gap between this volume-weighted value $r_{\text{vol}}$ and the median $p_{50}$ is a deliberate heterogeneity signal, so both are reported. Assets with a missing or zero size are excluded from the computation. A Dandiset with no usable asset reports empty values for all six fields.

pyproject.toml

@@ -15,7 +15,7 @@ allow-direct-references = true
 
 [project]
 name = "dandi-s3-log-extraction"
-version="1.0.2"
+version="1.0.3"
 authors = [
   { name="Cody Baker", email="cody.c.baker.phd@gmail.com" },
 ]
@@ -37,6 +37,7 @@ license = {file = "LICENSE.txt"}
 requires-python = ">=3.14"
 dependencies = [
     "beartype",
+    "numpy",
     "pandas",
     "tqdm",
     "PyYAML",

src/dandi_s3_log_extraction/_command_line_interface/_cli.py

@@ -8,7 +8,12 @@
 import s3_log_extraction
 
 from ..extractors import DandiRemoteS3LogAccessExtractor
-from ..summarize import generate_dandiset_summaries
+from ..summarize import (
+    generate_archive_summaries,
+    generate_archive_totals,
+    generate_dandiset_summaries,
+    generate_dandiset_totals,
+)
 
 
 # dandis3logextraction
@@ -231,7 +236,7 @@ def _update_summaries_cli(
     """Generate condensed summaries of activity."""
     match mode:
         case "archive":
-            s3_log_extraction.summarize.generate_archive_summaries(cache_directory=cache_directory)
+            generate_archive_summaries(cache_directory=cache_directory)
         case _:
             pick_as_list = pick.split(",") if pick is not None else None
             skip_as_list = skip.split(",") if skip is not None else None
@@ -244,3 +249,39 @@ def _update_summaries_cli(
                 unassociated=unassociated,
                 cache_directory=cache_directory,
             )
+
+
+# dandis3logextraction update totals
+@_update_cli.command(name="totals")
+@rich_click.option(
+    "--mode",
+    help=(
+        "Generate grand totals of activity across the extracted data. "
+        "Mode 'archive' aggregates over all Dandiset totals. "
+        "By default, per-Dandiset totals are generated."
+    ),
+    required=False,
+    type=rich_click.Choice(choices=["dandi", "archive"]),
+    default=None,
+)
+@rich_click.option(
+    "--cache",
+    "cache_directory",
+    help=(
+        "Path to the folder containing all previously extracted S3 access logs (`cache_directory`). "
+        "If not provided, the default cache directory from the configuration will be used."
+    ),
+    required=False,
+    type=rich_click.Path(file_okay=False, dir_okay=True),
+    default=None,
+)
+def _update_totals_cli(
+    mode: typing.Literal["dandi", "archive"] | None = None,
+    cache_directory: str | None = None,
+) -> None:
+    """Generate grand totals of all extracted data, including experimental delivery ratio fields."""
+    match mode:
+        case "archive":
+            generate_archive_totals(cache_directory=cache_directory)
+        case _:
+            generate_dandiset_totals(cache_directory=cache_directory)

src/dandi_s3_log_extraction/summarize/__init__.py

@@ -1,5 +1,13 @@
-from ._generate_dandiset_summaries import generate_dandiset_summaries
+from ._generate_dandiset_summaries import (
+    generate_archive_summaries,
+    generate_archive_totals,
+    generate_dandiset_summaries,
+    generate_dandiset_totals,
+)
 
 __all__ = [
+    "generate_archive_summaries",
+    "generate_archive_totals",
     "generate_dandiset_summaries",
+    "generate_dandiset_totals",
 ]