A real-time market gateway that ingests Hyperliquid data from Kafka, warms caches from Redis, streams low-latency snapshots over WebSockets, and exposes operational health/metrics endpoints.
- Overview
- Architecture
- Component Reference
- Startup Sequence
- Data Flow Patterns
- Configuration
- API Surface
- Development
- Docker & Deployment
- Monitoring & Observability
- Troubleshooting
- Performance Notes
- Design Decisions
- Future Improvements
- Related Services
- Contributing & Support
The gateway is the bridge between upstream market data sources and client-facing experiences:
- Consume Kafka topics for prices, depth, trades, overview, candles, and stats.
- Maintain in-memory market, overview, and stats stores with Redis-backed hydration to ensure warm start-ups.
- Stream live updates via a WebSocket fan-out server and provide REST APIs for snapshots and metadata.
- Resolve asset icons/logos through a multi-provider registry and cache the results.
- Continuously publish Prometheus metrics and expose
/healthzand/readyzendpoints for orchestration.
flowchart LR
Kafka[(Kafka Topics)] -->|prices/orderbook/candles/trades| Consumers
Consumers -->|upserts| Stores[(In-memory Stores)]
Stores -->|fan-out| WS[WebSocket Server]
Stores -->|snapshots| HTTP[HTTP API]
Stores -->|persist| Redis[(Redis Cache)]
Redis -->|hydrate| Stores
StatsRedis[(Stats Redis)] -->|hydrate/pubsub| Stores
Stores -->|metrics| PromMetrics[(Prometheus /metrics)]
- Event-driven ingestion – Kafka consumers parse and normalize upstream payloads (prices, orderbooks, trades, candles, metadata, BBO) before writing into the market store.
- Warm start hydration – On boot the service hydrates market, overview, and candle snapshots from Redis and fetches Hyperliquid’s asset universe to seed candle backfills.
- In-memory stores with Redis persistence – The
MarketStore,OverviewStore, andStatsStorekeep low-latency data in process while Redis keeps shared snapshots for new replicas. - Dual-server front-end – HTTP exposes read APIs plus
/healthz,/readyz,/metrics; WebSocket broadcasts incremental updates and optional stats channels. - Fail-fast watchdogs – Kafka fail-fast timers and managed shutdowns prevent serving stale data.
if (config.redis) {
redisClient = createManagedRedisClient({
url: config.redis.url,
name: 'gateway-redis',
logger,
onStateChange: (state: ManagedRedisState) => {
health.redis.connected = state.connected;
health.redis.lastError = state.lastError;
setRedisConnected(state.connected);
healthTracker.push();
}
});
}
const iconRegistry = new IconRegistry({
logger,
redis: redisClient
});
const hydratedAssets = await hydrateSnapshotsFromRedis({
redis: redisClient,
providerIds,
resolveProviderRedisKeyConfig,
marketStore,
overviewStore,
iconRegistry,
logger
});
await hydrateCandlesFromRedis({
redis: redisClient,
providerAssets,
providerIds,
resolveProviderRedisKeyConfig,
marketStore,
logger
});
const { httpServer, cleanup: cleanupHttpServer } = createHttpServer(...);
const { wsServer, broadcast, broadcastOverview, broadcastStatsAssets, broadcastStatsOverview, cleanup: cleanupWebSocket } =
attachWebSocketServer(httpServer, marketStore, overviewStore, logger, config.statsRedis ? statsStore : undefined, {
providerId: defaultProvider
}); const overviewFailFastWatchdog = createFailFastWatchdog({
timeoutMs: kafkaFailFastMs,
logger,
onTimeout: (reason) => {
logger.error({ reason, timeoutMs: kafkaFailFastMs }, 'Kafka overview fail-fast watchdog triggered');
handleFatalKafka(reason);
}
});
const marketConsumer = createMarketConsumer(
config,
logger,
(update, domains, providerHint) => {
const snapshot = marketStore.upsert(update, providerHint ?? defaultProvider);
if (snapshot) {
broadcast(snapshot, domains);
setSnapshotAge(snapshot.provider ?? defaultProvider ?? 'unknown', snapshot);
}
...
},
{
allowedProviders: providerIds,
defaultProvider,
failFast: {
timeoutMs: kafkaFailFastMs,
onTimeout: (reason) => handleFatalKafka(reason)
},
onFatalError: (error, context) => {
handleFatalKafka(context.reason, error);
},
lifecycle: { ... }
}
);The icon registry enriches asset metadata through a deterministic pipeline: explicit branding URLs, curated override lists, CDN symbol fallbacks, and finally a rate-limited CoinGecko search client. Results are cached in memory and optionally Redis with quota-aware lookup queues, while repeated provider failures are suppressed to trace-level noise.
export class IconRegistry {
private readonly redis?: Redis;
private readonly hitTtlSeconds: number;
private readonly missTtlSeconds: number;
private readonly assetCache = new Map<string, string | null>();
private readonly symbolCache = new Map<string, string | null>();
private readonly pendingAssets = new Map<string, Promise<string | null>>();
...
constructor(options: IconRegistryOptions) {
this.logger = options.logger.child({ module: 'iconRegistry' });
this.redis = options.redis;
this.hitTtlSeconds = Math.max(options.hitTtlSeconds ?? DEFAULT_HIT_TTL_SECONDS, 60);
this.missTtlSeconds = Math.max(options.missTtlSeconds ?? DEFAULT_MISS_TTL_SECONDS, 60);
this.maxConcurrentLookups = Math.max(1, options.maxConcurrentLookups ?? DEFAULT_MAX_CONCURRENT_LOOKUPS);
...
}The companion CoinGeckoClient (services/gateway/src/branding/coinGeckoClient.ts) memoizes search responses, enforces a small concurrency window with cooldowns after HTTP 429 responses, and supports the optional COINGECKO_API_KEY header for elevated quotas.
hyperliquid.ts– authenticated REST wrapper for spot/perp actions.hyperliquidCandles.ts– fetches historical candles used for hydration/backfill.hyperliquidUniverse.ts– discovers the live trading universe before Kafka traffic is available.
const DEFAULT_INFO_URL = process.env.HYPERLIQUID_INFO_URL ?? 'https://api.hyperliquid.xyz/info';
...
export const fetchHyperliquidUniverseAssets = async (
options: HyperliquidUniverseOptions = {}
): Promise<string[]> => {
const infoUrl = options.infoUrl?.trim().length ? options.infoUrl : DEFAULT_INFO_URL;
const payload = { type: 'metaAndAssetCtxs', dex: typeof options.dex === 'string' ? options.dex : '' };
...
const { statusCode, body } = await request(infoUrl, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(payload) });
if (statusCode !== 200) {
return [];
}
const text = await body.text();
if (!text) {
return [];
}
return normaliseAssetList(JSON.parse(text) as MetaAndCtxResponse);
};The Redis helpers (redis/candles.ts, redis/leaderboard.ts, redis/snapshots.ts, redis/stats.ts) encapsulate key derivation, snapshot serialization, and TTL management for each data type. They are consumed by runtime/hydration.ts, runtime/candles.ts, and runtime/stats.ts during runtime.
export const hydrateSnapshotsFromRedis = async ({
redis,
providerIds,
resolveProviderRedisKeyConfig,
marketStore,
overviewStore,
iconRegistry,
logger
}: HydrationDeps): Promise<ProviderAssetMap> => {
if (!redis) {
return new Map();
}
for (const providerId of providerIds) {
const keyConfig = resolveProviderRedisKeyConfig(providerId);
if (!keyConfig) {
logger.warn({ provider: providerId }, 'Skipping Redis hydration - missing key config');
continue;
}
const snapshots = await fetchLatestSnapshots(redis, keyConfig, providerId);
marketStore.hydrateFromRedis(snapshots, providerId, (snapshot) => {
...
setSnapshotAge(providerForSnapshot, snapshot);
...
});
const overviewRecord = await fetchOverviewRecord(redis, keyConfig, providerId);
overviewStore.hydrate(overviewRecord, providerId);
}hydration.ts– boot-time snapshot + overview hydration from Redis (see snippet above).candles.ts– candle hydration/backfill orchestrator with per-interval limits, small batches, and persistence back to Redis.health.ts&readiness.ts– shared gateway health snapshot plus readiness rules.stats.ts– attaches to a stats Redis namespace (dual connection) for periodic refreshes and pub/sub driven pushes to clients.
export const backfillHistoricalCandles = async ({
providerIds,
marketStore,
logger,
redis,
resolveProviderRedisKeyConfig,
intervals = INITIAL_BACKFILL_INTERVALS,
resolveBackfillAssets
}: CandleBackfillOptions): Promise<void> => {
const MAX_ASSETS_PER_BATCH = 5;
for (const providerId of providerIds) {
if (!isHyperliquidProvider(providerId)) {
continue;
}
const assets = await resolveAssetsForProvider({ providerId, marketStore, resolveBackfillAssets });
if (assets.length === 0) {
logger.debug({ provider: providerId }, 'Skipping candle backfill - no assets available yet');
continue;
}
for (let i = 0; i < assets.length; i += MAX_ASSETS_PER_BATCH) {
const batch = assets.slice(i, i + MAX_ASSETS_PER_BATCH);
for (const asset of batch) {
await backfillAssetCandles({ provider: providerId, asset, marketStore, logger, redis, resolveProviderRedisKeyConfig, intervals });
await new Promise((resolve) => setTimeout(resolve, 200));
}
if (i + MAX_ASSETS_PER_BATCH < assets.length) {
await new Promise((resolve) => setTimeout(resolve, 500));
}
}
}
};server/http.ts wires the Express server for health/readiness, metrics, and data APIs. server/websocket.ts hooks onto the HTTP server at /ws and manages subscription state, chunked snapshots, stats metadata, and ping/pong keep-alives.
export const attachWebSocketServer = (
httpServer: HttpServer,
marketStore: MarketStore,
overviewStore: OverviewStore,
logger: pino.Logger,
statsStore?: StatsStore,
options?: { providerId: string }
) => {
const wsServer = new WebSocketServer({ server: httpServer, path: '/ws' });
const subscriptions = new Map<WebSocket, Subscription>();
...
const emitMarketSnapshots = (socket: WebSocket, provider: string, snapshots: MarketSnapshot[]) => {
if (snapshots.length <= MARKET_SNAPSHOT_CHUNK_SIZE) {
socket.send(serialiseSnapshot(provider, snapshots));
return;
}
const totalChunks = Math.ceil(snapshots.length / MARKET_SNAPSHOT_CHUNK_SIZE);
for (let index = 0; index < totalChunks; index += 1) {
const start = index * MARKET_SNAPSHOT_CHUNK_SIZE;
const end = Math.min(start + MARKET_SNAPSHOT_CHUNK_SIZE, snapshots.length);
socket.send(serialiseSnapshot(provider, snapshots.slice(start, end), index + 1, totalChunks));
}
};
...
socket.on('message', (raw) => {
const message = JSON.parse(raw.toString());
if (message.type === 'subscribe') {
const domains = normaliseDomains(message.domains);
const assets = normaliseAssets(message.assets);
...
socket.send(JSON.stringify({ type: 'subscription', domains: domains ? Array.from(domains) : null, assets: assets ? Array.from(assets) : null, provider: subscription.provider }));
sendSnapshot(socket, subscription);
return;
}
if (message.type === 'ping') {
socket.send(JSON.stringify({ type: 'pong', at: Date.now() }));
return;
}
socket.send(JSON.stringify({ type: 'error', error: 'unknown_message_type' }));
});MarketStore, OverviewStore, and StatsStore hold canonical in-memory state, track TTLs, and generate derived artifacts like price history, trade tape, candles, and BBO spreads.
export class MarketStore {
private readonly store = new Map<string, Map<string, InternalState>>();
private readonly history = new Map<string, Map<string, PriceHistoryPoint[]>>();
private readonly trades = new Map<string, Map<string, TradeSummary[]>>();
private readonly candles = new Map<string, Map<string, Map<number, CandleSnapshot[]>>>();
...
upsert(
update: {
price?: PriceEvent;
orderbook?: OrderbookEvent;
metadata?: AssetMetadataEvent;
bbo?: BboEvent;
trade?: TradeEvent;
candle?: CandleSnapshotEvent;
},
fallbackProvider?: string
): MarketSnapshot | null {
const asset =
update.price?.asset ??
update.orderbook?.asset ??
update.metadata?.asset ??
update.bbo?.asset ??
update.trade?.asset ??
update.candle?.asset;
const provider =
update.price?.provider ??
update.orderbook?.provider ??
update.metadata?.provider ??
update.bbo?.provider ??
update.trade?.provider ??
update.candle?.provider ??
fallbackProvider ??
this.defaultProvider;
...
const snapshot = this.toSnapshot(provider, asset, current);
...
return snapshot;
}Kafka consumers are built on @pmon/core-service helpers. The market consumer registers all configured topics, parses messages per domain, enforces payload size limits, and dispatches normalized updates to the stores.
for (const topic of config.kafka.topics.orderbook) {
registerTopic(topic, 'order book', (payload) => {
const orderbookEvent = parseOrderbookEvent(payload);
if (!orderbookEvent) return null;
const priceEvent = derivePriceEvent(orderbookEvent);
return { orderbook: orderbookEvent, price: priceEvent ?? undefined };
});
}
...
const managedConsumer: ManagedKafkaConsumer = createManagedKafkaConsumer({
kafka: {
clientId: config.kafka.clientId,
brokers: config.kafka.brokers,
groupId: config.kafka.groupId
},
topics: Array.from(topicProcessors.keys()),
logger,
failFast: options.failFast,
onFatalError: options.onFatalError,
eachMessage: async ({ message, partition, topic }) => {
const value = message.value?.toString();
if (!value) return;
if (value.length > MAX_KAFKA_MESSAGE_SIZE) {
logger.warn({ topic, partition, size: value.length }, 'Kafka message too large, ignoring');
return;
}
const processor = topicProcessors.get(topic);
...
onEvent(update, domains, providerHint);
},
lifecycle
});utils/providers.ts centralizes provider tagging helpers used across hydration, stats, and candle backfills.
export const isHyperliquidProvider = (providerId?: string): boolean =>
typeof providerId === 'string' && providerId.toLowerCase().includes('hyperliquid');- Configuration & manifest selection –
loadConfig()merges env vars, ingestion manifests, and provider metadata, building topic lists and Redis key configs. - Runtime bootstrap –
runServicefrom@pmon/core-servicesets up logging, metrics registry, signal handlers, and health tracking. - Redis + icon registry – Redis clients are created (market + stats) and the icon registry is seeded to hydrate branding cache.
- State hydration – Market, overview, and candle snapshots are fetched from Redis and used to warm in-memory stores; asset universes are fetched via Hyperliquid’s REST API.
- Server bring-up – HTTP + WebSocket servers attach to the warmed stores before Kafka consumers start.
- Kafka & stats loops – Market + overview consumers join consumer groups, stats runtime subscribes to pub/sub and kicks off periodic refresh timers.
- Background maintenance – Stale snapshot pruning, icon lookup cleanup, candle backfills, and candle fetch cleanup intervals run until shutdown.
Cold starts in disconnected environments (e.g., Docker Compose without Kafka/Redis traffic) can still expose meaningful API responses by enabling the built‑in snapshot fixture:
- Controlled by
GATEWAY_ENABLE_SNAPSHOT_FIXTURE(defaultfalse); set totrueexplicitly when you need demo filler data. GATEWAY_SNAPSHOT_FIXTURE_PATHpoints to a JSON array of seed entries (defaults tosrc/fixtures/hyperliquid-demo.json).- On boot, if no market snapshots are hydrated for the default provider, the fixture is loaded and written into the
MarketStore, provider asset registry, andOverviewStore. - The same pass now emits derived stats objects, so
/api/v1/stats/*and WebSocket stats channels have data even while Kafka/Redis are offline. GATEWAY_FIXTURE_FEE_RATE(default0.00035) controls the fee assumption used when deriving 24h fee estimates from the fixture’s reported volume.- If stale snapshot pruning ever drains the store (e.g., no live Kafka traffic for hours), the gateway automatically rehydrates from the fixture after
GATEWAY_FIXTURE_RESEED_COOLDOWN_MS(default 10 minutes) so dashboards don’t stay empty.
This keeps UI dashboards (markets, open interest, overview summaries) functional while other services are down, and removes the need for ad-hoc scripts whenever the gateway restarts on a laptop or CI runner.
- Kafka publishes order book/price/metadata/trade/candle/BBO events.
createMarketConsumerparses each payload, derives price midpoints when needed, filters by provider, and emits normalized updates.MarketStoreupserts each asset snapshot, records price history, and triggers on-demand candle backfills if needed.- WebSocket broadcast fans out the updated snapshot to subscribed clients and updates snapshot-age metrics.
- Redis holds the latest canonical snapshot set for each provider.
- On startup,
hydrateSnapshotsFromRedisloads the dataset into the market/overview stores and records which assets are hydrated per provider. MarketStoreperiodically prunes stale assets (default 30 min) and rehydrates from Redis when needed.- Candle histories hydrate from Redis and are backfilled from Hyperliquid REST when missing, then persisted back to Redis for the next cold start.
- Incoming metadata events trigger
IconRegistry.resolveIcon. - The registry checks Redis/in-memory caches first, then tries explicit branding URLs, override lists, CDN fallbacks, and finally upstream APIs.
- Results (hit or miss) are cached with tiered TTLs to reduce API pressure; concurrency and queue length limits prevent unbounded lookups.
Environment variables are centralized in src/config.ts with manifest-aware defaults.
export const loadConfig = async (): Promise<GatewayConfig> => {
const kafkaBrokers = normaliseList(process.env.KAFKA_BROKER);
...
return {
manifestPath,
port: Number(process.env.PORT ?? 4000),
kafka: {
brokers: kafkaBrokers,
clientId: process.env.KAFKA_CLIENT_ID ?? 'pmon-gateway',
groupId: process.env.KAFKA_CONSUMER_GROUP ?? 'pmon-gateway',
topics: mergedTopics
},
redis,
statsRedis,
providers,
providerConfigs,
selection: {
providerId: selection?.provider.id ?? (manifest?.providers[0]?.id ?? 'hyperliquid'),
providerLabel: selection?.provider.displayName ?? selection?.provider.provider,
adapterId: selection?.adapter.id ?? selection?.adapter.kind ?? 'hyperliquid-perp',
shard: selection?.runtime.shard ?? {
id: `env:${parseNonNegativeInt(process.env.HYPERLIQUID_SHARD_INDEX) ?? 0}`,
index: parseNonNegativeInt(process.env.HYPERLIQUID_SHARD_INDEX) ?? 0,
count: parsePositiveInt(process.env.HYPERLIQUID_SHARD_COUNT) ?? 1
},
metadata: {
topicPrefix: selection?.provider.topicPrefix ?? DEFAULT_TOPIC_PREFIX,
redisNamespace: redis?.keyConfig.namespace ?? selection?.provider.redisNamespace,
tags: selection?.provider.tags
}
}
};
};| Variable | Default | Purpose |
|---|---|---|
PORT |
4000 |
HTTP/WebSocket port. |
KAFKA_BROKER |
(required) | Comma-delimited broker list consumed by normaliseList. |
KAFKA_CLIENT_ID / KAFKA_CONSUMER_GROUP |
pmon-gateway |
Kafka identifiers for market streams. |
KAFKA_TOPIC_* |
hyperliquid.<domain> |
Per-domain fallback topics when a manifest isn’t provided. |
INGEST_MANIFEST_PATH |
auto-detect configs/ingest*.yaml |
Path to ingestion manifest for selecting provider/adapters/shards. |
INGEST_PROVIDER_ID, INGEST_ADAPTER_ID, INGEST_SHARD_ID, INGEST_SHARD_INDEX |
hyperliquid / hyperliquid-perp |
Hint which provider/shard entry to use. |
HYPERLIQUID_SHARD_INDEX, HYPERLIQUID_SHARD_COUNT |
0 / 1 |
Fallback shard metadata when manifest hints are absent. |
REDIS_URL, REDIS_NAMESPACE, REDIS_TTL_SECONDS |
unset / pmon:hyperliquid:markets |
Redis hydration connection/key space. |
STATS_REDIS_URL, STATS_REDIS_NAMESPACE, STATS_REDIS_ASSET_PATTERN, STATS_REDIS_OVERVIEW_KEY |
unset / pmon:stats:metrics |
Dedicated Redis instance for stats runtime + pub/sub channel names. |
KAFKA_FAIL_FAST_TIMEOUT_MS |
45000 |
Fail-fast watchdog threshold for Kafka disconnections. |
GATEWAY_ENABLE_STARTUP_CANDLE_BACKFILL, ENABLE_STARTUP_CANDLE_BACKFILL, STARTUP_CANDLE_BACKFILL |
false |
Toggle initial candle backfill prior to serving traffic. |
GATEWAY_ENABLE_SNAPSHOT_FIXTURE |
false |
Seed market + stats stores from GATEWAY_SNAPSHOT_FIXTURE_PATH when no live data is available (opt-in for demos only). |
GATEWAY_SNAPSHOT_FIXTURE_PATH |
src/fixtures/hyperliquid-demo.json |
Relative/absolute path to the snapshot seed file. |
GATEWAY_FIXTURE_FEE_RATE |
0.00035 |
Fee assumption used when deriving 24h fee estimates from fixtures. |
GATEWAY_FIXTURE_RESEED_COOLDOWN_MS |
600000 |
Minimum delay before the gateway auto-reseeds fixtures after snapshots age out. |
GATEWAY_PROVIDER_ASSET_CAP |
1500 |
Bounded size for each provider’s tracked asset set (prevents runaway universes). |
GATEWAY_PROVIDER_ASSET_TTL_MINUTES |
360 |
TTL for provider asset observations used by backfill + pruning logic. |
GATEWAY_PROVIDER_ASSET_PRUNE_INTERVAL_MINUTES |
10 |
Frequency for trimming provider asset registries and backfill state. |
GATEWAY_ICON_MAX_CONCURRENCY, GATEWAY_ICON_MAX_QUEUE_SIZE, GATEWAY_ICON_QUEUE_WAIT_MS |
12 / 2000 / 15000 | Icon lookup throttling controls. |
GATEWAY_MEMORY_WARN_PCT, GATEWAY_MEMORY_CRITICAL_PCT, GATEWAY_MEMORY_FORCE_GC_PCT |
80 / 92 / 88 | Heap-utilization thresholds that drive logging, GC hints, and throttling. |
GATEWAY_ENABLE_HEAPDUMP_ENDPOINT |
false |
When true, enables POST /admin/heapdump for on-demand .heapsnapshot capture (only recommended in trusted environments). |
GATEWAY_MAX_WEBSOCKET_BUFFER_BYTES |
5_000_000 |
Disconnect WebSocket clients whose bufferedAmount exceeds this threshold to prevent runaway heap usage. |
GATEWAY_MAX_WEBSOCKET_SNAPSHOT_ASSETS |
500 |
Upper bound on the number of markets included in a single snapshot payload (applies to initial hydration + assets=null subscriptions). |
GATEWAY_DEFAULT_WS_DOMAINS |
all |
Comma-separated set of domains auto-subscribed for new WebSocket clients (e.g. prices,orderbook,metadata). |
HYPERLIQUID_INFO_URL |
https://api.hyperliquid.xyz/info |
Override for Hyperliquid info endpoint used during backfill. |
TRADER_INTEL_REDIS_URL, TRADER_INTEL_REDIS_KEY_PREFIX, APP_URL |
unset | Optional admin endpoints for trader-intelligence cache controls. |
GET /healthz– liveness payload containing derived BBO ages per provider.GET /readyz– readiness decision (503 when Kafka/Redis dependencies are unavailable).GET /metrics– Prometheus exposition format.GET /api/v1/markets– latest market snapshots for the default provider (aliases exist per provider).GET /api/v1/markets/:assetplus subpaths for metadata, price history, candles, trades, orderbook depth, BBO.GET /api/v1/overview– provider overview snapshot.
const buildHealthSnapshot = () => {
const health = getHealth();
...
const evaluation = evaluateGatewayReadiness(healthWithDerived);
return { health: healthWithDerived, evaluation };
};
app.get('/healthz', (_req, res) => {
const { health, evaluation } = buildHealthSnapshot();
res.status(200).json({
status: 'ok',
reasons: fatalReasons.length > 0 ? fatalReasons : undefined,
warnings: warnings.length > 0 ? warnings : undefined,
health
});
});
app.get('/readyz', (_req, res) => {
const { health, evaluation } = buildHealthSnapshot();
res.status(ready ? 200 : 503).json({
status: ready ? 'ready' : 'degraded',
reasons: ready ? undefined : fatalReasons,
warnings: warnings.length > 0 ? warnings : undefined,
health
});
});
app.get('/api/v1/markets', (_req, res) => {
if (!defaultProvider) {
res.status(400).json({ error: 'provider_not_configured' });
return;
}
const data: MarketSnapshot[] = marketStore.snapshot(defaultProvider);
res.json({ data });
});
app.get('/api/v1/markets/:asset', (req, res) => {
...
});- Connect to
ws://<host>:<port>/ws. - Initial connection delivers chunked
snapshotframes plusoverview-updateand optionalstats-snapshotframes. - Clients subscribe with
{ "type": "subscribe", "provider": "hyperliquid", "domains": ["prices", "stats"], "assets": ["BTC", "ETH"] }. - Messages emitted:
snapshot– batched market snapshots with optional chunk metadata.market-update– incremental update containingdomainstouched.overview-update,stats-snapshot,stats-asset-update,stats-overview-update.subscriptionack,pongreplies, and structurederrorframes for invalid messages.
Use the workspace scripts for local development:
"scripts": {
"build": "tsc -p tsconfig.json",
"start": "node dist/index.js",
"dev": "tsx watch src/index.ts",
"smoke": "tsx scripts/smoke.ts",
"clean": "rimraf dist tsconfig.tsbuildinfo"
},# Install dependencies
yarn install
# Run with hot reload (tsx)
yarn workspace @pmon/gateway dev
# Type-check + build
yarn workspace @pmon/gateway build
yarn workspace @pmon/gateway start
# Smoke test HTTP + WS endpoints against local stack
yarn workspace @pmon/gateway smokeThe multi-stage Dockerfile builds shared workspaces (core-service, ingestion-sdk, trader-intelligence) before producing a lean Node 20 runtime image.
FROM node:20-slim AS base
WORKDIR /app
...
FROM deps AS dev
ENV NODE_ENV=development
ENV NODE_OPTIONS=--max-old-space-size=2048
...
FROM base AS runner
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/services/gateway/dist ./services/gateway/dist
EXPOSE 4000
CMD ["node", "services/gateway/dist/index.js"]
# Build and run locally
docker build -t gateway-service services/gateway
docker run -p 4000:4000 \
-e KAFKA_BROKER=localhost:9092 \
-e REDIS_URL=redis://redis:6379 \
gateway-service
# Compose (if docker-compose.yml defines the gateway service)
docker-compose up -d gatewayPrometheus metrics (prefixed gateway_) are registered via prom-client.
collectDefaultMetrics({ register: registry, prefix: 'gateway_' });
const redisConnectedGauge = new Gauge({
name: 'gateway_redis_connected',
help: 'Redis connection state (1 = connected, 0 = disconnected)',
registers: [registry]
});
const bboTimestampGauge = new Gauge({
name: 'gateway_hyperliquid_bbo_timestamp_ms',
help: 'Unix timestamp in milliseconds of the last Hyperliquid BBO update observed by the gateway',
registers: [registry]
});
const redisSnapshotAgeGauge = new Gauge({
name: 'gateway_redis_snapshot_age_ms',
help: 'Age in milliseconds of the latest cached market snapshot per asset',
labelNames: ['provider', 'asset'],
registers: [registry]
});Key metrics
gateway_redis_connected– Redis availability.gateway_hyperliquid_bbo_timestamp_ms&gateway_hyperliquid_bbo_age_ms– freshness of best-bid-offer stream.gateway_redis_snapshot_age_ms{provider,asset}– staleness watchdog for each market snapshot.gateway_websocket_backpressure_drops_total{context}– number of WebSocket clients disconnected because their send buffer exceededGATEWAY_MAX_WEBSOCKET_BUFFER_BYTES.- Default process metrics from
collectDefaultMetrics(heap, CPU, GC).
- The Helm chart exposes
ingestor.reliability.*inkubernetes/chart/pmon/values.yaml, which are rendered into the deployment as:INGEST_KAFKA_OVERLOAD_BACKOFF_MS– base backoff (ms) between Kafka producer retries.INGEST_KAFKA_OVERLOAD_MAX_RETRIES– number of retry attempts before surfacing the error to the adapter.INGEST_KAFKA_OVERLOAD_LOG_INTERVAL_MS– minimum interval between “Kafka producer overload detected” log entries to keep noise manageable.INGEST_PUBLISH_THROTTLE_RATIO,INGEST_PUBLISH_THROTTLE_DELAY_MS,INGEST_PUBLISH_THROTTLE_MAX_WAIT_MS– thresholds used by the worker to pause publishes whenpendingPublishesapproachesINGEST_MAX_PENDING_PUBLISHES.
- These env vars can also be set directly when running the ingestor outside Helm; the worker reads them on boot and exposes the following Prometheus signals:
ingestion_worker_kafka_overload_events_total– number of observed timeout/backpressure events.ingestion_worker_kafka_overload_logs_suppressed_total– how many overload logs were suppressed by the rate limiter.ingestion_worker_publish_throttle_total+ingestion_worker_publish_throttle_duration_ms– how often the worker had to pause and for how long.ingestion_worker_publish_backpressure_total– counter for requests rejected because the pending queue never drained beforeINGEST_MAX_PENDING_PUBLISHES.ingestion_worker_kafka_retry_total– number of adapter level retries (labelled by topic + error type) which is a useful early warning that brokers are struggling.
Health signals
/healthz– always 200 but includes fatal reasons + warnings for observability./readyz– 200 only when Redis and Kafka market consumer are ready; warns when overview stream is degraded.
- Runtime diagnostics – The memory monitor logs heap usage every 30 seconds along with
providerAssetsTotal, icon registry queues, stats cache sizes, and more. - Backpressure guard –
GATEWAY_MAX_WEBSOCKET_BUFFER_BYTESdrops slow clients before they exhaust the heap; monitorgateway_websocket_backpressure_drops_total. - Snapshot limits –
GATEWAY_MAX_WEBSOCKET_SNAPSHOT_ASSETScaps the number of markets included per subscription to keep snapshot frames predictable. - Domain defaults – Use
GATEWAY_DEFAULT_WS_DOMAINSto restrict which feeds are auto-subscribed, reducing the amount of data broadcast to every connection. - On-demand heap dumps – When
GATEWAY_ENABLE_HEAPDUMP_ENDPOINT=true,POST /admin/heapdumpwrites/tmp/gateway-heap-<ts>.heapsnapshot; copy it out (docker cp) and inspect with Chrome DevTools or VS Code. - See
docs/operations/runbooks/gateway-memory.mdfor detailed procedures.
Logs
- Startup/shutdown lifecycle, Kafka consumer state transitions, Redis connection changes.
- Icon lookup failures (rate limits, queue overflow), snapshot pruning, candle backfill progress.
- Backpressure drops (
Dropping WebSocket client due to send backlog) and heap snapshot captures.
Although the gateway runs well on a laptop + docker compose, the majority of the heavy lifting happens in long‑lived Kubernetes clusters. Keep the following in mind when promoting changes:
- Memory guarantees – Allocate at least 1.5 GB of container memory to leave headroom for Node’s default 3 GB heap (or explicitly set
NODE_OPTIONS="--max-old-space-size=<MB>"). Pair withresources.requests/limitsso the pod isn’t OOM‑killed by the kubelet. - Ephemeral storage – Heap dumps and temporary fixture files are written to
/tmp. Mount an emptyDir with sufficient space or disable heap dumps (GATEWAY_ENABLE_HEAPDUMP_ENDPOINT=false) in production. - Process signals – The service traps
SIGTERM/SIGINT. Give the pod a generousterminationGracePeriodSeconds(≥ 30 s) so Kafka consumers can commit offsets and Redis connections can flush state.
- Horizontal sharding – Add more replicas by giving each deployment a unique shard or provider via the manifest envs (
INGEST_PROVIDER_ID,INGEST_SHARD_ID, etc.). Thekubernetes/chart/pmontemplates already expose these knobs. - Autoscaling – An HPA based on CPU alone is usually sufficient, but you can also target
gateway_websocket_connectionsor Kafka consumer lag via custom metrics. - Backpressure alerts – Page when
gateway_websocket_backpressure_drops_totalexceeds a threshold over a 5‑minute window; that’s the best leading indicator of overloaded clients before heap usage rises.
- Dashboards – Plot
gateway_heap_utilization_pct,gateway_websocket_backpressure_drops_total, Kafka lag, and Redis snapshot ages in the same Grafana dashboard for a single view of health. - Tracing & logs – The service emits structured JSON logs compatible with Loki/Datadog. Configure your log pipeline to index
module,provider, andlabelfields for quick triage. - Readiness gates – Wire
/readyzinto ingress controllers or service meshes (Istio/Linkerd) to avoid routing traffic to pods that are still hydrating from Redis.
- Build and push the Docker image (
docker buildx build --pushor use the provided CI pipeline). - Update the Helm release or Kubernetes manifests with the new tag and any env changes (e.g.
GATEWAY_DEFAULT_WS_DOMAINS, memory thresholds). - Verify the pod reaches Ready state and emits healthy
memory-monitorlogs. - Run smoke tests against
/api/v1/markets,/ws, and/metrics. - Observe dashboards for at least one backfill cycle to ensure snapshot pruning, icon lookup queues, and Kafka throughput look normal.
These guardrails help keep the gateway predictable as traffic, asset universe size, and client behavior evolve in real production clusters.
gateway.resources– set CPU/memory requests & limits to guarantee scheduling and prevent noisy-neighbor preemption (e.g. requests:cpu: 500m,memory: 1Gi; limits:cpu: 2,memory: 4Gi).gateway.extraEnv– pass through environment overrides such asGATEWAY_DEFAULT_WS_DOMAINSor heap thresholds.gateway.hpa.*– enable the bundled HorizontalPodAutoscaler (autoscaling/v2) to scale replicas automatically:
gateway:
hpa:
enabled: true
minReplicas: 2
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 65You can extend metrics with custom or external Signals (Kafka lag, WebSocket drops, etc.) to align with your SLOs.
| Symptom | Likely Cause | Mitigation |
|---|---|---|
| Heap out-of-memory | Oversized candle/history retention or icon lookups. | Reduce snapshot retention (MarketStore.pruneAllStaleSnapshots already runs every 5 minutes), tune GATEWAY_ICON_* envs, raise NODE_OPTIONS=--max-old-space-size=<MB>. |
| Kafka consumer lag | Slow processing or broker instability. | Inspect logs for fail-fast watchdog triggers, scale out consumer replicas (distinct shard IDs), verify Redis isn’t blocking event loop. |
| WebSocket disconnects | Network churn or oversized payloads. | Confirm chunk sizes (MARKET_SNAPSHOT_CHUNK_SIZE = 75 assets) are sufficient, enforce client-side ping/pong, review gateway_websocket_connections metric. |
| Redis hydration skips | Missing key configs per provider. | Ensure manifests define redisNamespace per shard or set REDIS_NAMESPACE/REDIS_URL. |
- Memory – Baseline ~200 MB, climbing to ~800 MB with 1000 assets + candle histories; icon cache adds another 100–200 MB. Each WebSocket subscription adds ~50 KB.
- Throughput – Designed for >10 k WebSocket broadcasts/s, ~5 k Kafka messages/s, ~20 k Redis ops/s.
- Latency – WebSocket fan-out <5 ms (p99). Kafka-to-client propagation <10 ms (p99) when brokers keep up. Snapshot fetches from Redis/hydrated store <2 ms (p99).
- In-memory first – Memory-resident stores guarantee sub-millisecond reads; Redis keeps replicas warm.
- Stateful gateway – Maintaining snapshots (vs stateless proxies) enables derived metrics, price-history, and BBO age tracking.
- Dual Redis channels – Market snapshots and stats metrics use separate Redis connections/namespaces to avoid contention.
- Lazy icon lookup – On-demand lookups with caching avoid blocking the ingestion path while still delivering rich UI assets.
- Fail-fast Kafka watchdogs – Proactively exit when no fresh data is flowing instead of serving stale quotes.
- Bounded pruning – Scheduled pruning/backfill loops prevent unbounded memory growth.
- Horizontal scaling with Redis-backed state reconciliation.
- Grafana dashboards for gateway metrics + Kafka lag.
- Bulk icon lookup optimizations (batched provider requests).
- WebSocket authentication & rate limiting.
- Historical replay / time-travel debugging endpoint.
- Memory-friendly compression for large snapshots.
- Circuit breakers for third-party APIs (icon providers, Hyperliquid REST).
- Ingestor – Produces the Kafka topics consumed here; manifests align shards and namespaces.
- Stats service – Publishes derived metrics into the stats Redis namespace consumed by
StatsStore. - Trader Intelligence – Optional admin endpoints in
server/http.tsintegrate with trader data caches. - Frontend/UI – Subscribes to WebSocket channels and hits
/api/v1/*endpoints for hydration.
Refer to docs/apis/gateway.md and docs/providers/hyperliquid.md for cross-service contracts.
- Create feature branches, run
yarn workspace @pmon/gateway devfor iterative work. - Before opening a PR, run
yarn workspace @pmon/gateway buildandyarn workspace @pmon/gateway smoke. - Keep docs in
docs/apis/gateway.mdin sync when API surface changes. - For questions ping the
#gateway-serviceSlack channel or open an issue in this repository.
Happy shipping!