Skip to content

Latest commit

 

History

History
834 lines (709 loc) · 39.9 KB

README.md

File metadata and controls

834 lines (709 loc) · 39.9 KB

Gateway Service

A real-time market gateway that ingests Hyperliquid data from Kafka, warms caches from Redis, streams low-latency snapshots over WebSockets, and exposes operational health/metrics endpoints.

Table of Contents

Overview

The gateway is the bridge between upstream market data sources and client-facing experiences:

  • Consume Kafka topics for prices, depth, trades, overview, candles, and stats.
  • Maintain in-memory market, overview, and stats stores with Redis-backed hydration to ensure warm start-ups.
  • Stream live updates via a WebSocket fan-out server and provide REST APIs for snapshots and metadata.
  • Resolve asset icons/logos through a multi-provider registry and cache the results.
  • Continuously publish Prometheus metrics and expose /healthz and /readyz endpoints for orchestration.

Architecture

Data Flow

flowchart LR
  Kafka[(Kafka Topics)] -->|prices/orderbook/candles/trades| Consumers
  Consumers -->|upserts| Stores[(In-memory Stores)]
  Stores -->|fan-out| WS[WebSocket Server]
  Stores -->|snapshots| HTTP[HTTP API]
  Stores -->|persist| Redis[(Redis Cache)]
  Redis -->|hydrate| Stores
  StatsRedis[(Stats Redis)] -->|hydrate/pubsub| Stores
  Stores -->|metrics| PromMetrics[(Prometheus /metrics)]
Loading

Operational Patterns

  • Event-driven ingestion – Kafka consumers parse and normalize upstream payloads (prices, orderbooks, trades, candles, metadata, BBO) before writing into the market store.
  • Warm start hydration – On boot the service hydrates market, overview, and candle snapshots from Redis and fetches Hyperliquid’s asset universe to seed candle backfills.
  • In-memory stores with Redis persistence – The MarketStore, OverviewStore, and StatsStore keep low-latency data in process while Redis keeps shared snapshots for new replicas.
  • Dual-server front-end – HTTP exposes read APIs plus /healthz, /readyz, /metrics; WebSocket broadcasts incremental updates and optional stats channels.
  • Fail-fast watchdogs – Kafka fail-fast timers and managed shutdowns prevent serving stale data.
  if (config.redis) {
    redisClient = createManagedRedisClient({
      url: config.redis.url,
      name: 'gateway-redis',
      logger,
      onStateChange: (state: ManagedRedisState) => {
        health.redis.connected = state.connected;
        health.redis.lastError = state.lastError;
        setRedisConnected(state.connected);
        healthTracker.push();
      }
    });
  }

  const iconRegistry = new IconRegistry({
    logger,
    redis: redisClient
  });

  const hydratedAssets = await hydrateSnapshotsFromRedis({
    redis: redisClient,
    providerIds,
    resolveProviderRedisKeyConfig,
    marketStore,
    overviewStore,
    iconRegistry,
    logger
  });

  await hydrateCandlesFromRedis({
    redis: redisClient,
    providerAssets,
    providerIds,
    resolveProviderRedisKeyConfig,
    marketStore,
    logger
  });

  const { httpServer, cleanup: cleanupHttpServer } = createHttpServer(...);
  const { wsServer, broadcast, broadcastOverview, broadcastStatsAssets, broadcastStatsOverview, cleanup: cleanupWebSocket } =
    attachWebSocketServer(httpServer, marketStore, overviewStore, logger, config.statsRedis ? statsStore : undefined, {
      providerId: defaultProvider
    });
  const overviewFailFastWatchdog = createFailFastWatchdog({
    timeoutMs: kafkaFailFastMs,
    logger,
    onTimeout: (reason) => {
      logger.error({ reason, timeoutMs: kafkaFailFastMs }, 'Kafka overview fail-fast watchdog triggered');
      handleFatalKafka(reason);
    }
  });

  const marketConsumer = createMarketConsumer(
    config,
    logger,
    (update, domains, providerHint) => {
      const snapshot = marketStore.upsert(update, providerHint ?? defaultProvider);
      if (snapshot) {
        broadcast(snapshot, domains);
        setSnapshotAge(snapshot.provider ?? defaultProvider ?? 'unknown', snapshot);
      }
      ...
    },
    {
      allowedProviders: providerIds,
      defaultProvider,
      failFast: {
        timeoutMs: kafkaFailFastMs,
        onTimeout: (reason) => handleFatalKafka(reason)
      },
      onFatalError: (error, context) => {
        handleFatalKafka(context.reason, error);
      },
      lifecycle: { ... }
    }
  );

Component Reference

branding

The icon registry enriches asset metadata through a deterministic pipeline: explicit branding URLs, curated override lists, CDN symbol fallbacks, and finally a rate-limited CoinGecko search client. Results are cached in memory and optionally Redis with quota-aware lookup queues, while repeated provider failures are suppressed to trace-level noise.

export class IconRegistry {
  private readonly redis?: Redis;
  private readonly hitTtlSeconds: number;
  private readonly missTtlSeconds: number;
  private readonly assetCache = new Map<string, string | null>();
  private readonly symbolCache = new Map<string, string | null>();
  private readonly pendingAssets = new Map<string, Promise<string | null>>();
  ...
  constructor(options: IconRegistryOptions) {
    this.logger = options.logger.child({ module: 'iconRegistry' });
    this.redis = options.redis;
    this.hitTtlSeconds = Math.max(options.hitTtlSeconds ?? DEFAULT_HIT_TTL_SECONDS, 60);
    this.missTtlSeconds = Math.max(options.missTtlSeconds ?? DEFAULT_MISS_TTL_SECONDS, 60);
    this.maxConcurrentLookups = Math.max(1, options.maxConcurrentLookups ?? DEFAULT_MAX_CONCURRENT_LOOKUPS);
    ...
  }

The companion CoinGeckoClient (services/gateway/src/branding/coinGeckoClient.ts) memoizes search responses, enforces a small concurrency window with cooldowns after HTTP 429 responses, and supports the optional COINGECKO_API_KEY header for elevated quotas.

clients

  • hyperliquid.ts – authenticated REST wrapper for spot/perp actions.
  • hyperliquidCandles.ts – fetches historical candles used for hydration/backfill.
  • hyperliquidUniverse.ts – discovers the live trading universe before Kafka traffic is available.
const DEFAULT_INFO_URL = process.env.HYPERLIQUID_INFO_URL ?? 'https://api.hyperliquid.xyz/info';
...
export const fetchHyperliquidUniverseAssets = async (
  options: HyperliquidUniverseOptions = {}
): Promise<string[]> => {
  const infoUrl = options.infoUrl?.trim().length ? options.infoUrl : DEFAULT_INFO_URL;
  const payload = { type: 'metaAndAssetCtxs', dex: typeof options.dex === 'string' ? options.dex : '' };
  ...
  const { statusCode, body } = await request(infoUrl, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(payload) });
  if (statusCode !== 200) {
    return [];
  }
  const text = await body.text();
  if (!text) {
    return [];
  }
  return normaliseAssetList(JSON.parse(text) as MetaAndCtxResponse);
};

redis

The Redis helpers (redis/candles.ts, redis/leaderboard.ts, redis/snapshots.ts, redis/stats.ts) encapsulate key derivation, snapshot serialization, and TTL management for each data type. They are consumed by runtime/hydration.ts, runtime/candles.ts, and runtime/stats.ts during runtime.

export const hydrateSnapshotsFromRedis = async ({
  redis,
  providerIds,
  resolveProviderRedisKeyConfig,
  marketStore,
  overviewStore,
  iconRegistry,
  logger
}: HydrationDeps): Promise<ProviderAssetMap> => {
  if (!redis) {
    return new Map();
  }
  for (const providerId of providerIds) {
    const keyConfig = resolveProviderRedisKeyConfig(providerId);
    if (!keyConfig) {
      logger.warn({ provider: providerId }, 'Skipping Redis hydration - missing key config');
      continue;
    }
    const snapshots = await fetchLatestSnapshots(redis, keyConfig, providerId);
    marketStore.hydrateFromRedis(snapshots, providerId, (snapshot) => {
      ...
      setSnapshotAge(providerForSnapshot, snapshot);
      ...
    });
    const overviewRecord = await fetchOverviewRecord(redis, keyConfig, providerId);
    overviewStore.hydrate(overviewRecord, providerId);
  }

runtime

  • hydration.ts – boot-time snapshot + overview hydration from Redis (see snippet above).
  • candles.ts – candle hydration/backfill orchestrator with per-interval limits, small batches, and persistence back to Redis.
  • health.ts & readiness.ts – shared gateway health snapshot plus readiness rules.
  • stats.ts – attaches to a stats Redis namespace (dual connection) for periodic refreshes and pub/sub driven pushes to clients.
export const backfillHistoricalCandles = async ({
  providerIds,
  marketStore,
  logger,
  redis,
  resolveProviderRedisKeyConfig,
  intervals = INITIAL_BACKFILL_INTERVALS,
  resolveBackfillAssets
}: CandleBackfillOptions): Promise<void> => {
  const MAX_ASSETS_PER_BATCH = 5;
  for (const providerId of providerIds) {
    if (!isHyperliquidProvider(providerId)) {
      continue;
    }
    const assets = await resolveAssetsForProvider({ providerId, marketStore, resolveBackfillAssets });
    if (assets.length === 0) {
      logger.debug({ provider: providerId }, 'Skipping candle backfill - no assets available yet');
      continue;
    }
    for (let i = 0; i < assets.length; i += MAX_ASSETS_PER_BATCH) {
      const batch = assets.slice(i, i + MAX_ASSETS_PER_BATCH);
      for (const asset of batch) {
        await backfillAssetCandles({ provider: providerId, asset, marketStore, logger, redis, resolveProviderRedisKeyConfig, intervals });
        await new Promise((resolve) => setTimeout(resolve, 200));
      }
      if (i + MAX_ASSETS_PER_BATCH < assets.length) {
        await new Promise((resolve) => setTimeout(resolve, 500));
      }
    }
  }
};

server

server/http.ts wires the Express server for health/readiness, metrics, and data APIs. server/websocket.ts hooks onto the HTTP server at /ws and manages subscription state, chunked snapshots, stats metadata, and ping/pong keep-alives.

export const attachWebSocketServer = (
  httpServer: HttpServer,
  marketStore: MarketStore,
  overviewStore: OverviewStore,
  logger: pino.Logger,
  statsStore?: StatsStore,
  options?: { providerId: string }
) => {
  const wsServer = new WebSocketServer({ server: httpServer, path: '/ws' });
  const subscriptions = new Map<WebSocket, Subscription>();
  ...
  const emitMarketSnapshots = (socket: WebSocket, provider: string, snapshots: MarketSnapshot[]) => {
    if (snapshots.length <= MARKET_SNAPSHOT_CHUNK_SIZE) {
      socket.send(serialiseSnapshot(provider, snapshots));
      return;
    }
    const totalChunks = Math.ceil(snapshots.length / MARKET_SNAPSHOT_CHUNK_SIZE);
    for (let index = 0; index < totalChunks; index += 1) {
      const start = index * MARKET_SNAPSHOT_CHUNK_SIZE;
      const end = Math.min(start + MARKET_SNAPSHOT_CHUNK_SIZE, snapshots.length);
      socket.send(serialiseSnapshot(provider, snapshots.slice(start, end), index + 1, totalChunks));
    }
  };
  ...
  socket.on('message', (raw) => {
    const message = JSON.parse(raw.toString());
    if (message.type === 'subscribe') {
      const domains = normaliseDomains(message.domains);
      const assets = normaliseAssets(message.assets);
      ...
      socket.send(JSON.stringify({ type: 'subscription', domains: domains ? Array.from(domains) : null, assets: assets ? Array.from(assets) : null, provider: subscription.provider }));
      sendSnapshot(socket, subscription);
      return;
    }
    if (message.type === 'ping') {
      socket.send(JSON.stringify({ type: 'pong', at: Date.now() }));
      return;
    }
    socket.send(JSON.stringify({ type: 'error', error: 'unknown_message_type' }));
  });

store

MarketStore, OverviewStore, and StatsStore hold canonical in-memory state, track TTLs, and generate derived artifacts like price history, trade tape, candles, and BBO spreads.

export class MarketStore {
  private readonly store = new Map<string, Map<string, InternalState>>();
  private readonly history = new Map<string, Map<string, PriceHistoryPoint[]>>();
  private readonly trades = new Map<string, Map<string, TradeSummary[]>>();
  private readonly candles = new Map<string, Map<string, Map<number, CandleSnapshot[]>>>();
  ...
  upsert(
    update: {
      price?: PriceEvent;
      orderbook?: OrderbookEvent;
      metadata?: AssetMetadataEvent;
      bbo?: BboEvent;
      trade?: TradeEvent;
      candle?: CandleSnapshotEvent;
    },
    fallbackProvider?: string
  ): MarketSnapshot | null {
    const asset =
      update.price?.asset ??
      update.orderbook?.asset ??
      update.metadata?.asset ??
      update.bbo?.asset ??
      update.trade?.asset ??
      update.candle?.asset;
    const provider =
      update.price?.provider ??
      update.orderbook?.provider ??
      update.metadata?.provider ??
      update.bbo?.provider ??
      update.trade?.provider ??
      update.candle?.provider ??
      fallbackProvider ??
      this.defaultProvider;
    ...
    const snapshot = this.toSnapshot(provider, asset, current);
    ...
    return snapshot;
  }

streams

Kafka consumers are built on @pmon/core-service helpers. The market consumer registers all configured topics, parses messages per domain, enforces payload size limits, and dispatches normalized updates to the stores.

for (const topic of config.kafka.topics.orderbook) {
  registerTopic(topic, 'order book', (payload) => {
    const orderbookEvent = parseOrderbookEvent(payload);
    if (!orderbookEvent) return null;
    const priceEvent = derivePriceEvent(orderbookEvent);
    return { orderbook: orderbookEvent, price: priceEvent ?? undefined };
  });
}
...
const managedConsumer: ManagedKafkaConsumer = createManagedKafkaConsumer({
  kafka: {
    clientId: config.kafka.clientId,
    brokers: config.kafka.brokers,
    groupId: config.kafka.groupId
  },
  topics: Array.from(topicProcessors.keys()),
  logger,
  failFast: options.failFast,
  onFatalError: options.onFatalError,
  eachMessage: async ({ message, partition, topic }) => {
    const value = message.value?.toString();
    if (!value) return;
    if (value.length > MAX_KAFKA_MESSAGE_SIZE) {
      logger.warn({ topic, partition, size: value.length }, 'Kafka message too large, ignoring');
      return;
    }
    const processor = topicProcessors.get(topic);
    ...
    onEvent(update, domains, providerHint);
  },
  lifecycle
});

utils

utils/providers.ts centralizes provider tagging helpers used across hydration, stats, and candle backfills.

export const isHyperliquidProvider = (providerId?: string): boolean =>
  typeof providerId === 'string' && providerId.toLowerCase().includes('hyperliquid');

Startup Sequence

  1. Configuration & manifest selection – loadConfig() merges env vars, ingestion manifests, and provider metadata, building topic lists and Redis key configs.
  2. Runtime bootstrap – runService from @pmon/core-service sets up logging, metrics registry, signal handlers, and health tracking.
  3. Redis + icon registry – Redis clients are created (market + stats) and the icon registry is seeded to hydrate branding cache.
  4. State hydration – Market, overview, and candle snapshots are fetched from Redis and used to warm in-memory stores; asset universes are fetched via Hyperliquid’s REST API.
  5. Server bring-up – HTTP + WebSocket servers attach to the warmed stores before Kafka consumers start.
  6. Kafka & stats loops – Market + overview consumers join consumer groups, stats runtime subscribes to pub/sub and kicks off periodic refresh timers.
  7. Background maintenance – Stale snapshot pruning, icon lookup cleanup, candle backfills, and candle fetch cleanup intervals run until shutdown.

Snapshot Fixture Mode

Cold starts in disconnected environments (e.g., Docker Compose without Kafka/Redis traffic) can still expose meaningful API responses by enabling the built‑in snapshot fixture:

  • Controlled by GATEWAY_ENABLE_SNAPSHOT_FIXTURE (default false); set to true explicitly when you need demo filler data.
  • GATEWAY_SNAPSHOT_FIXTURE_PATH points to a JSON array of seed entries (defaults to src/fixtures/hyperliquid-demo.json).
  • On boot, if no market snapshots are hydrated for the default provider, the fixture is loaded and written into the MarketStore, provider asset registry, and OverviewStore.
  • The same pass now emits derived stats objects, so /api/v1/stats/* and WebSocket stats channels have data even while Kafka/Redis are offline.
  • GATEWAY_FIXTURE_FEE_RATE (default 0.00035) controls the fee assumption used when deriving 24h fee estimates from the fixture’s reported volume.
  • If stale snapshot pruning ever drains the store (e.g., no live Kafka traffic for hours), the gateway automatically rehydrates from the fixture after GATEWAY_FIXTURE_RESEED_COOLDOWN_MS (default 10 minutes) so dashboards don’t stay empty.

This keeps UI dashboards (markets, open interest, overview summaries) functional while other services are down, and removes the need for ad-hoc scripts whenever the gateway restarts on a laptop or CI runner.

Data Flow Patterns

Real-Time Market Data

  1. Kafka publishes order book/price/metadata/trade/candle/BBO events.
  2. createMarketConsumer parses each payload, derives price midpoints when needed, filters by provider, and emits normalized updates.
  3. MarketStore upserts each asset snapshot, records price history, and triggers on-demand candle backfills if needed.
  4. WebSocket broadcast fans out the updated snapshot to subscribed clients and updates snapshot-age metrics.

Snapshot Lifecycle

  1. Redis holds the latest canonical snapshot set for each provider.
  2. On startup, hydrateSnapshotsFromRedis loads the dataset into the market/overview stores and records which assets are hydrated per provider.
  3. MarketStore periodically prunes stale assets (default 30 min) and rehydrates from Redis when needed.
  4. Candle histories hydrate from Redis and are backfilled from Hyperliquid REST when missing, then persisted back to Redis for the next cold start.

Icon Lookup

  1. Incoming metadata events trigger IconRegistry.resolveIcon.
  2. The registry checks Redis/in-memory caches first, then tries explicit branding URLs, override lists, CDN fallbacks, and finally upstream APIs.
  3. Results (hit or miss) are cached with tiered TTLs to reduce API pressure; concurrency and queue length limits prevent unbounded lookups.

Configuration

Environment variables are centralized in src/config.ts with manifest-aware defaults.

export const loadConfig = async (): Promise<GatewayConfig> => {
  const kafkaBrokers = normaliseList(process.env.KAFKA_BROKER);
  ...
  return {
    manifestPath,
    port: Number(process.env.PORT ?? 4000),
    kafka: {
      brokers: kafkaBrokers,
      clientId: process.env.KAFKA_CLIENT_ID ?? 'pmon-gateway',
      groupId: process.env.KAFKA_CONSUMER_GROUP ?? 'pmon-gateway',
      topics: mergedTopics
    },
    redis,
    statsRedis,
    providers,
    providerConfigs,
    selection: {
      providerId: selection?.provider.id ?? (manifest?.providers[0]?.id ?? 'hyperliquid'),
      providerLabel: selection?.provider.displayName ?? selection?.provider.provider,
      adapterId: selection?.adapter.id ?? selection?.adapter.kind ?? 'hyperliquid-perp',
      shard: selection?.runtime.shard ?? {
        id: `env:${parseNonNegativeInt(process.env.HYPERLIQUID_SHARD_INDEX) ?? 0}`,
        index: parseNonNegativeInt(process.env.HYPERLIQUID_SHARD_INDEX) ?? 0,
        count: parsePositiveInt(process.env.HYPERLIQUID_SHARD_COUNT) ?? 1
      },
      metadata: {
        topicPrefix: selection?.provider.topicPrefix ?? DEFAULT_TOPIC_PREFIX,
        redisNamespace: redis?.keyConfig.namespace ?? selection?.provider.redisNamespace,
        tags: selection?.provider.tags
      }
    }
  };
};
Variable Default Purpose
PORT 4000 HTTP/WebSocket port.
KAFKA_BROKER (required) Comma-delimited broker list consumed by normaliseList.
KAFKA_CLIENT_ID / KAFKA_CONSUMER_GROUP pmon-gateway Kafka identifiers for market streams.
KAFKA_TOPIC_* hyperliquid.<domain> Per-domain fallback topics when a manifest isn’t provided.
INGEST_MANIFEST_PATH auto-detect configs/ingest*.yaml Path to ingestion manifest for selecting provider/adapters/shards.
INGEST_PROVIDER_ID, INGEST_ADAPTER_ID, INGEST_SHARD_ID, INGEST_SHARD_INDEX hyperliquid / hyperliquid-perp Hint which provider/shard entry to use.
HYPERLIQUID_SHARD_INDEX, HYPERLIQUID_SHARD_COUNT 0 / 1 Fallback shard metadata when manifest hints are absent.
REDIS_URL, REDIS_NAMESPACE, REDIS_TTL_SECONDS unset / pmon:hyperliquid:markets Redis hydration connection/key space.
STATS_REDIS_URL, STATS_REDIS_NAMESPACE, STATS_REDIS_ASSET_PATTERN, STATS_REDIS_OVERVIEW_KEY unset / pmon:stats:metrics Dedicated Redis instance for stats runtime + pub/sub channel names.
KAFKA_FAIL_FAST_TIMEOUT_MS 45000 Fail-fast watchdog threshold for Kafka disconnections.
GATEWAY_ENABLE_STARTUP_CANDLE_BACKFILL, ENABLE_STARTUP_CANDLE_BACKFILL, STARTUP_CANDLE_BACKFILL false Toggle initial candle backfill prior to serving traffic.
GATEWAY_ENABLE_SNAPSHOT_FIXTURE false Seed market + stats stores from GATEWAY_SNAPSHOT_FIXTURE_PATH when no live data is available (opt-in for demos only).
GATEWAY_SNAPSHOT_FIXTURE_PATH src/fixtures/hyperliquid-demo.json Relative/absolute path to the snapshot seed file.
GATEWAY_FIXTURE_FEE_RATE 0.00035 Fee assumption used when deriving 24h fee estimates from fixtures.
GATEWAY_FIXTURE_RESEED_COOLDOWN_MS 600000 Minimum delay before the gateway auto-reseeds fixtures after snapshots age out.
GATEWAY_PROVIDER_ASSET_CAP 1500 Bounded size for each provider’s tracked asset set (prevents runaway universes).
GATEWAY_PROVIDER_ASSET_TTL_MINUTES 360 TTL for provider asset observations used by backfill + pruning logic.
GATEWAY_PROVIDER_ASSET_PRUNE_INTERVAL_MINUTES 10 Frequency for trimming provider asset registries and backfill state.
GATEWAY_ICON_MAX_CONCURRENCY, GATEWAY_ICON_MAX_QUEUE_SIZE, GATEWAY_ICON_QUEUE_WAIT_MS 12 / 2000 / 15000 Icon lookup throttling controls.
GATEWAY_MEMORY_WARN_PCT, GATEWAY_MEMORY_CRITICAL_PCT, GATEWAY_MEMORY_FORCE_GC_PCT 80 / 92 / 88 Heap-utilization thresholds that drive logging, GC hints, and throttling.
GATEWAY_ENABLE_HEAPDUMP_ENDPOINT false When true, enables POST /admin/heapdump for on-demand .heapsnapshot capture (only recommended in trusted environments).
GATEWAY_MAX_WEBSOCKET_BUFFER_BYTES 5_000_000 Disconnect WebSocket clients whose bufferedAmount exceeds this threshold to prevent runaway heap usage.
GATEWAY_MAX_WEBSOCKET_SNAPSHOT_ASSETS 500 Upper bound on the number of markets included in a single snapshot payload (applies to initial hydration + assets=null subscriptions).
GATEWAY_DEFAULT_WS_DOMAINS all Comma-separated set of domains auto-subscribed for new WebSocket clients (e.g. prices,orderbook,metadata).
HYPERLIQUID_INFO_URL https://api.hyperliquid.xyz/info Override for Hyperliquid info endpoint used during backfill.
TRADER_INTEL_REDIS_URL, TRADER_INTEL_REDIS_KEY_PREFIX, APP_URL unset Optional admin endpoints for trader-intelligence cache controls.

API Surface

HTTP Endpoints

  • GET /healthz – liveness payload containing derived BBO ages per provider.
  • GET /readyz – readiness decision (503 when Kafka/Redis dependencies are unavailable).
  • GET /metrics – Prometheus exposition format.
  • GET /api/v1/markets – latest market snapshots for the default provider (aliases exist per provider).
  • GET /api/v1/markets/:asset plus subpaths for metadata, price history, candles, trades, orderbook depth, BBO.
  • GET /api/v1/overview – provider overview snapshot.
  const buildHealthSnapshot = () => {
    const health = getHealth();
    ...
    const evaluation = evaluateGatewayReadiness(healthWithDerived);
    return { health: healthWithDerived, evaluation };
  };

  app.get('/healthz', (_req, res) => {
    const { health, evaluation } = buildHealthSnapshot();
    res.status(200).json({
      status: 'ok',
      reasons: fatalReasons.length > 0 ? fatalReasons : undefined,
      warnings: warnings.length > 0 ? warnings : undefined,
      health
    });
  });

  app.get('/readyz', (_req, res) => {
    const { health, evaluation } = buildHealthSnapshot();
    res.status(ready ? 200 : 503).json({
      status: ready ? 'ready' : 'degraded',
      reasons: ready ? undefined : fatalReasons,
      warnings: warnings.length > 0 ? warnings : undefined,
      health
    });
  });

  app.get('/api/v1/markets', (_req, res) => {
    if (!defaultProvider) {
      res.status(400).json({ error: 'provider_not_configured' });
      return;
    }
    const data: MarketSnapshot[] = marketStore.snapshot(defaultProvider);
    res.json({ data });
  });

  app.get('/api/v1/markets/:asset', (req, res) => {
    ...
  });

WebSocket Protocol

  • Connect to ws://<host>:<port>/ws.
  • Initial connection delivers chunked snapshot frames plus overview-update and optional stats-snapshot frames.
  • Clients subscribe with { "type": "subscribe", "provider": "hyperliquid", "domains": ["prices", "stats"], "assets": ["BTC", "ETH"] }.
  • Messages emitted:
    • snapshot – batched market snapshots with optional chunk metadata.
    • market-update – incremental update containing domains touched.
    • overview-update, stats-snapshot, stats-asset-update, stats-overview-update.
    • subscription ack, pong replies, and structured error frames for invalid messages.

Development

Use the workspace scripts for local development:

  "scripts": {
    "build": "tsc -p tsconfig.json",
    "start": "node dist/index.js",
    "dev": "tsx watch src/index.ts",
    "smoke": "tsx scripts/smoke.ts",
    "clean": "rimraf dist tsconfig.tsbuildinfo"
  },
# Install dependencies
yarn install

# Run with hot reload (tsx)
yarn workspace @pmon/gateway dev

# Type-check + build
yarn workspace @pmon/gateway build
yarn workspace @pmon/gateway start

# Smoke test HTTP + WS endpoints against local stack
yarn workspace @pmon/gateway smoke

Docker & Deployment

The multi-stage Dockerfile builds shared workspaces (core-service, ingestion-sdk, trader-intelligence) before producing a lean Node 20 runtime image.

FROM node:20-slim AS base
WORKDIR /app
...
FROM deps AS dev
ENV NODE_ENV=development
ENV NODE_OPTIONS=--max-old-space-size=2048
...
FROM base AS runner
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/services/gateway/dist ./services/gateway/dist
EXPOSE 4000
CMD ["node", "services/gateway/dist/index.js"]
# Build and run locally
docker build -t gateway-service services/gateway
docker run -p 4000:4000 \
  -e KAFKA_BROKER=localhost:9092 \
  -e REDIS_URL=redis://redis:6379 \
  gateway-service

# Compose (if docker-compose.yml defines the gateway service)
docker-compose up -d gateway

Monitoring & Observability

Prometheus metrics (prefixed gateway_) are registered via prom-client.

collectDefaultMetrics({ register: registry, prefix: 'gateway_' });
const redisConnectedGauge = new Gauge({
  name: 'gateway_redis_connected',
  help: 'Redis connection state (1 = connected, 0 = disconnected)',
  registers: [registry]
});
const bboTimestampGauge = new Gauge({
  name: 'gateway_hyperliquid_bbo_timestamp_ms',
  help: 'Unix timestamp in milliseconds of the last Hyperliquid BBO update observed by the gateway',
  registers: [registry]
});
const redisSnapshotAgeGauge = new Gauge({
  name: 'gateway_redis_snapshot_age_ms',
  help: 'Age in milliseconds of the latest cached market snapshot per asset',
  labelNames: ['provider', 'asset'],
  registers: [registry]
});

Key metrics

  • gateway_redis_connected – Redis availability.
  • gateway_hyperliquid_bbo_timestamp_ms & gateway_hyperliquid_bbo_age_ms – freshness of best-bid-offer stream.
  • gateway_redis_snapshot_age_ms{provider,asset} – staleness watchdog for each market snapshot.
  • gateway_websocket_backpressure_drops_total{context} – number of WebSocket clients disconnected because their send buffer exceeded GATEWAY_MAX_WEBSOCKET_BUFFER_BYTES.
  • Default process metrics from collectDefaultMetrics (heap, CPU, GC).

Hyperliquid ingestor reliability knobs

  • The Helm chart exposes ingestor.reliability.* in kubernetes/chart/pmon/values.yaml, which are rendered into the deployment as:
    • INGEST_KAFKA_OVERLOAD_BACKOFF_MS – base backoff (ms) between Kafka producer retries.
    • INGEST_KAFKA_OVERLOAD_MAX_RETRIES – number of retry attempts before surfacing the error to the adapter.
    • INGEST_KAFKA_OVERLOAD_LOG_INTERVAL_MS – minimum interval between “Kafka producer overload detected” log entries to keep noise manageable.
    • INGEST_PUBLISH_THROTTLE_RATIO, INGEST_PUBLISH_THROTTLE_DELAY_MS, INGEST_PUBLISH_THROTTLE_MAX_WAIT_MS – thresholds used by the worker to pause publishes when pendingPublishes approaches INGEST_MAX_PENDING_PUBLISHES.
  • These env vars can also be set directly when running the ingestor outside Helm; the worker reads them on boot and exposes the following Prometheus signals:
    • ingestion_worker_kafka_overload_events_total – number of observed timeout/backpressure events.
    • ingestion_worker_kafka_overload_logs_suppressed_total – how many overload logs were suppressed by the rate limiter.
    • ingestion_worker_publish_throttle_total + ingestion_worker_publish_throttle_duration_ms – how often the worker had to pause and for how long.
    • ingestion_worker_publish_backpressure_total – counter for requests rejected because the pending queue never drained before INGEST_MAX_PENDING_PUBLISHES.
    • ingestion_worker_kafka_retry_total – number of adapter level retries (labelled by topic + error type) which is a useful early warning that brokers are struggling.

Health signals

  • /healthz – always 200 but includes fatal reasons + warnings for observability.
  • /readyz – 200 only when Redis and Kafka market consumer are ready; warns when overview stream is degraded.

Memory controls & heap dumps

  • Runtime diagnostics – The memory monitor logs heap usage every 30 seconds along with providerAssetsTotal, icon registry queues, stats cache sizes, and more.
  • Backpressure guard – GATEWAY_MAX_WEBSOCKET_BUFFER_BYTES drops slow clients before they exhaust the heap; monitor gateway_websocket_backpressure_drops_total.
  • Snapshot limits – GATEWAY_MAX_WEBSOCKET_SNAPSHOT_ASSETS caps the number of markets included per subscription to keep snapshot frames predictable.
  • Domain defaults – Use GATEWAY_DEFAULT_WS_DOMAINS to restrict which feeds are auto-subscribed, reducing the amount of data broadcast to every connection.
  • On-demand heap dumps – When GATEWAY_ENABLE_HEAPDUMP_ENDPOINT=true, POST /admin/heapdump writes /tmp/gateway-heap-<ts>.heapsnapshot; copy it out (docker cp) and inspect with Chrome DevTools or VS Code.
  • See docs/operations/runbooks/gateway-memory.md for detailed procedures.

Logs

  • Startup/shutdown lifecycle, Kafka consumer state transitions, Redis connection changes.
  • Icon lookup failures (rate limits, queue overflow), snapshot pruning, candle backfill progress.
  • Backpressure drops (Dropping WebSocket client due to send backlog) and heap snapshot captures.

Cloud Native & Production Considerations

Although the gateway runs well on a laptop + docker compose, the majority of the heavy lifting happens in long‑lived Kubernetes clusters. Keep the following in mind when promoting changes:

Container runtime

  • Memory guarantees – Allocate at least 1.5 GB of container memory to leave headroom for Node’s default 3 GB heap (or explicitly set NODE_OPTIONS="--max-old-space-size=<MB>"). Pair with resources.requests/limits so the pod isn’t OOM‑killed by the kubelet.
  • Ephemeral storage – Heap dumps and temporary fixture files are written to /tmp. Mount an emptyDir with sufficient space or disable heap dumps (GATEWAY_ENABLE_HEAPDUMP_ENDPOINT=false) in production.
  • Process signals – The service traps SIGTERM/SIGINT. Give the pod a generous terminationGracePeriodSeconds (≥ 30 s) so Kafka consumers can commit offsets and Redis connections can flush state.

Scaling & resiliency

  • Horizontal sharding – Add more replicas by giving each deployment a unique shard or provider via the manifest envs (INGEST_PROVIDER_ID, INGEST_SHARD_ID, etc.). The kubernetes/chart/pmon templates already expose these knobs.
  • Autoscaling – An HPA based on CPU alone is usually sufficient, but you can also target gateway_websocket_connections or Kafka consumer lag via custom metrics.
  • Backpressure alerts – Page when gateway_websocket_backpressure_drops_total exceeds a threshold over a 5‑minute window; that’s the best leading indicator of overloaded clients before heap usage rises.

Observability

  • Dashboards – Plot gateway_heap_utilization_pct, gateway_websocket_backpressure_drops_total, Kafka lag, and Redis snapshot ages in the same Grafana dashboard for a single view of health.
  • Tracing & logs – The service emits structured JSON logs compatible with Loki/Datadog. Configure your log pipeline to index module, provider, and label fields for quick triage.
  • Readiness gates – Wire /readyz into ingress controllers or service meshes (Istio/Linkerd) to avoid routing traffic to pods that are still hydrating from Redis.

Deployment checklist

  1. Build and push the Docker image (docker buildx build --push or use the provided CI pipeline).
  2. Update the Helm release or Kubernetes manifests with the new tag and any env changes (e.g. GATEWAY_DEFAULT_WS_DOMAINS, memory thresholds).
  3. Verify the pod reaches Ready state and emits healthy memory-monitor logs.
  4. Run smoke tests against /api/v1/markets, /ws, and /metrics.
  5. Observe dashboards for at least one backfill cycle to ensure snapshot pruning, icon lookup queues, and Kafka throughput look normal.

These guardrails help keep the gateway predictable as traffic, asset universe size, and client behavior evolve in real production clusters.

Helm / Kubernetes knobs

  • gateway.resources – set CPU/memory requests & limits to guarantee scheduling and prevent noisy-neighbor preemption (e.g. requests: cpu: 500m, memory: 1Gi; limits: cpu: 2, memory: 4Gi).
  • gateway.extraEnv – pass through environment overrides such as GATEWAY_DEFAULT_WS_DOMAINS or heap thresholds.
  • gateway.hpa.* – enable the bundled HorizontalPodAutoscaler (autoscaling/v2) to scale replicas automatically:
gateway:
  hpa:
    enabled: true
    minReplicas: 2
    maxReplicas: 8
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 65

You can extend metrics with custom or external Signals (Kafka lag, WebSocket drops, etc.) to align with your SLOs.

Troubleshooting

Symptom Likely Cause Mitigation
Heap out-of-memory Oversized candle/history retention or icon lookups. Reduce snapshot retention (MarketStore.pruneAllStaleSnapshots already runs every 5 minutes), tune GATEWAY_ICON_* envs, raise NODE_OPTIONS=--max-old-space-size=<MB>.
Kafka consumer lag Slow processing or broker instability. Inspect logs for fail-fast watchdog triggers, scale out consumer replicas (distinct shard IDs), verify Redis isn’t blocking event loop.
WebSocket disconnects Network churn or oversized payloads. Confirm chunk sizes (MARKET_SNAPSHOT_CHUNK_SIZE = 75 assets) are sufficient, enforce client-side ping/pong, review gateway_websocket_connections metric.
Redis hydration skips Missing key configs per provider. Ensure manifests define redisNamespace per shard or set REDIS_NAMESPACE/REDIS_URL.

Performance Notes

  • Memory – Baseline ~200 MB, climbing to ~800 MB with 1000 assets + candle histories; icon cache adds another 100–200 MB. Each WebSocket subscription adds ~50 KB.
  • Throughput – Designed for >10 k WebSocket broadcasts/s, ~5 k Kafka messages/s, ~20 k Redis ops/s.
  • Latency – WebSocket fan-out <5 ms (p99). Kafka-to-client propagation <10 ms (p99) when brokers keep up. Snapshot fetches from Redis/hydrated store <2 ms (p99).

Design Decisions

  1. In-memory first – Memory-resident stores guarantee sub-millisecond reads; Redis keeps replicas warm.
  2. Stateful gateway – Maintaining snapshots (vs stateless proxies) enables derived metrics, price-history, and BBO age tracking.
  3. Dual Redis channels – Market snapshots and stats metrics use separate Redis connections/namespaces to avoid contention.
  4. Lazy icon lookup – On-demand lookups with caching avoid blocking the ingestion path while still delivering rich UI assets.
  5. Fail-fast Kafka watchdogs – Proactively exit when no fresh data is flowing instead of serving stale quotes.
  6. Bounded pruning – Scheduled pruning/backfill loops prevent unbounded memory growth.

Future Improvements

  • Horizontal scaling with Redis-backed state reconciliation.
  • Grafana dashboards for gateway metrics + Kafka lag.
  • Bulk icon lookup optimizations (batched provider requests).
  • WebSocket authentication & rate limiting.
  • Historical replay / time-travel debugging endpoint.
  • Memory-friendly compression for large snapshots.
  • Circuit breakers for third-party APIs (icon providers, Hyperliquid REST).

Related Services

  • Ingestor – Produces the Kafka topics consumed here; manifests align shards and namespaces.
  • Stats service – Publishes derived metrics into the stats Redis namespace consumed by StatsStore.
  • Trader Intelligence – Optional admin endpoints in server/http.ts integrate with trader data caches.
  • Frontend/UI – Subscribes to WebSocket channels and hits /api/v1/* endpoints for hydration.

Refer to docs/apis/gateway.md and docs/providers/hyperliquid.md for cross-service contracts.

Contributing & Support

  1. Create feature branches, run yarn workspace @pmon/gateway dev for iterative work.
  2. Before opening a PR, run yarn workspace @pmon/gateway build and yarn workspace @pmon/gateway smoke.
  3. Keep docs in docs/apis/gateway.md in sync when API surface changes.
  4. For questions ping the #gateway-service Slack channel or open an issue in this repository.

Happy shipping!