Skip to content

Latest commit

 

History

History
146 lines (110 loc) · 8.39 KB

stats.md

File metadata and controls

146 lines (110 loc) · 8.39 KB

Stats Service (proposed)

Goal: ingest the raw market streams produced by the ingestor (Kafka/Redpanda topics) and maintain higher-level analytics that power richer UI visualizations and cross-exchange comparisons.

Stats & Analytics Layer

Derived analytics consolidate raw Hyperliquid and future exchange data into math-rich indicators surfaced across the platform.

Group Example Metrics
Funding & Carry Funding rate, funding volatility, z-score, carry yield
Leverage & Risk OI change, OI/Volume ratio, liquidation density, leverage pressure
Volatility & Regime Realized vol, volatility ratio, volatility–funding correlation
Order Flow & Liquidity Spread width, depth, imbalance, trade flow, impact
Cross-Exchange Comparisons Basis vs peers, funding difference, liquidity efficiency
Composite Indices Stress index, market stability, carry-momentum

Metrics include hover-ready descriptions (“why this number matters”) and inline formulas. Values flow Stats Service → Gateway caches → UI components to keep phrasing and math consistent across experiences. Funding analytics now include averages, medians, volatility, and crowding ratios for every provider.

Current metric coverage (Nov 2025)

Domain Fields emitted Quant intuition
Price dynamics last, simpleMovingAverage, realizedVolatility, longRealizedVolatility, volatilityRatio, skewness, kurtosis, windowSize SMA filters micro-structure noise while realised/long volatility and shape stats (skew/kurtosis) highlight regime shifts.
Funding & carry current, average, volatility, zScore, momentum, positiveShare, returnCorrelation, sampleCount Highlights crowding: dispersion and z-score call out extreme funding; correlation versus returns exposes directional coupling.
Leverage & risk openInterestUsd, previousOpenInterestUsd, openInterestDeltaUsd, openInterestDeltaPct, openInterestChange1m, openInterestChange5m, openInterestChange1h, volume24h, oiToVolumeRatio Captures leverage pressure via OI deltas across multiple horizons and OI-to-volume ratios that often precede squeezes.
Liquidity spreadAbs, spreadBps, spreadStdDev, topOfBookFrequency, topOfBookDepth, orderBookImbalance, priceImpactPerUnit, fillProbabilityEstimate, mid, source Spread levels, depth, imbalance, and execution estimates expose trading friction and directional pressure; update frequency highlights order book churn/staleness.
Order flow buyVolume, sellVolume, rollingVolume, rollingVolume1m, rollingVolume5m, rollingVolume1h, cumulativeVolumeDelta, tradeFlowImbalance, tradeCount, tradeCount1m, tradeCount5m, tradeCount1h Quantifies directional aggressor flow for momentum/CVD dashboards and heatmaps.

Redis surface & observability

  • Asset level: stats:asset:<provider>:<symbol> → DerivedAssetMetrics.
  • Provider overview: stats:overview:<provider> → aggregate averages/totals for dashboards.
  • TTL mirrors STATS_REDIS_TTL_SECONDS; absence means hot set persists while Kafka updates replace content.
  • Prometheus: gauges stats_funding_volatility, stats_spread_stddev_bps, and stats_open_interest_delta_usd surface rollups for alerting/graphing.

Gateway integration

  • Configure the gateway with STATS_REDIS_URL and STATS_REDIS_NAMESPACE (defaults to pmon:stats:metrics).
  • Optional overrides:
    • STATS_REDIS_ASSET_PATTERN (default asset:{provider}:*)
    • STATS_REDIS_OVERVIEW_KEY (default overview:{provider})
  • HTTP endpoints now include:
    • GET /api/v1/stats/overview
    • GET /api/v1/stats/:asset
    • GET /api/v1/providers/:provider/stats/overview
    • GET /api/v1/providers/:provider/stats/:asset
  • Stats data is refreshed periodically from Redis (15s interval) and exposed alongside existing market snapshots.

Deployment matrix

  • Docker Compose ships a stats service alongside the ingestors/gateway/UI. It mounts configs/ingest.sample.yaml, points at the composed Kafka/Redis instances, and exposes 4010 for /healthz + /metrics.
  • Helm / Argo CD: the chart now includes stats.* values (image, Kafka IDs, Redis namespace, window size). The gateway automatically reads STATS_REDIS_URL/STATS_REDIS_NAMESPACE so Web/API surfaces hydrate as soon as the stats pod warms caches.
  • Redis namespace defaults to the base pmon:stats:metrics. When STATS_APPEND_PROVIDER_NAMESPACE (Helm value stats.redis.appendProviderNamespace) is enabled and the manifest resolves a single provider, the service appends the provider ID (e.g. pmon:stats:metrics:hyperliquid). Gateway/UI deployments must mirror the same setting when populating STATS_REDIS_NAMESPACE; disable the flag (recommended for multi-provider manifests) to keep the base namespace and ensure pub/sub channels match.

Proposed responsibilities

  • Subscribe to order book, trade, funding, and overview topics for every provider.
  • Compute rolling metrics (moving averages, volatility, funding deltas, liquidity scores, etc.).
  • Emit derived datasets to Redis or dedicated Kafka topics so the gateway can expose lightweight APIs.
  • Surface per-asset/per-exchange comparisons for the UI (e.g., best spread across exchanges, volume dominance).

Backlog

The stats backlog now lives in roadmap/roadmap.md under Architecture & Cross-Cutting → Stats Service.

Metrics, Aggregations, and Derived Data (new)

Funding & Carry

  • Funding rate average
  • Funding rate median
  • Funding rate standard deviation / volatility
  • Funding rate z-score
  • Cross-exchange funding difference
  • Annualized carry yield calculation
  • Rolling funding volatility (short window = analytics window)
  • Funding rate momentum (change over time)
  • Net funding bias (% positive vs negative) — surfaced via positiveShare

Leverage & Risk

  • Open interest (OI) delta and rate of change
  • OI / Volume ratio
  • OI / Market Cap ratio
  • Cumulative OI change chart data
  • Liquidation density and concentration estimates
  • Leverage pressure index = OI × Funding / Market Cap
  • System stress metric = correlation(OIΔ, priceΔ)
  • Total notional exposure per asset and overall

Volatility & Regime

  • Realized volatility (single configurable window; multi-horizon pending)
  • Rolling volatility ratio (short / long)
  • Volatility of funding rates
  • Volatility–funding correlation
  • Skew and kurtosis of returns
  • Regime label (calm, trending, squeeze, volatile)

Order-Flow & Liquidity

  • Spread width (best ask − best bid)
  • Spread standard deviation (spread volatility)
  • Average market depth within ±X% of mid (top-of-book proxy)
  • Order book imbalance (bid vs ask volume)
  • Cumulative volume delta (aggressive buy − aggressive sell)
  • Trade flow imbalance over rolling window
  • Price impact curve (Δprice / Δvolume proxy)
  • Fill probability estimate = depth × volatility factor
  • Top-of-book update frequency

Cross-Exchange Comparisons

  • Price difference vs other exchanges
  • Funding difference vs other exchanges
  • Volume share and dominance
  • Latency differential (timestamp skew)
  • Liquidity efficiency ratio (depthHL / depthPeers)

Composite & Summary Indices

  • Stress index combining OIΔ, funding, and volatility
  • Carry–momentum composite (funding × return)
  • Market stability index = 1 / (vol × fundingVol × OIΔ)
  • Entropy of funding rates across assets
  • Breadth metrics (% assets up / down)
  • Average and median metric summaries per category

Implementation TODOs

  • Persist derived values to Redis (stats:asset:*, stats:overview:*)
  • Expose /api/v1/stats/overview and /api/v1/stats/:asset
  • Include field metadata: displayName, formula, unit, description
  • Schedule rolling computations at configurable intervals (streaming window)
  • Add Prometheus metrics for funding_volatility, spread_stddev, OI_delta
  • Integrate with UI via gateway caches (stats:overview, stats:asset:*)

Open questions

  • Should analytics run continuously (streaming) or via scheduled batch jobs?
  • Do we need historical storage beyond Redis (e.g., BigQuery, Cloud Storage) for long-term charts?
  • How will we expose metrics to external consumers (REST, websockets, data dumps)?

Keep this doc updated as the design solidifies. Once implementation starts, the service should follow the same release workflow (make cut-release) and be managed via the Argo CD chart alongside other services.