Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs-mintlify/reference/configuration/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1385,6 +1385,31 @@ this variable.
| ------------------------- | ---------------------- | --------------------- |
| A valid number in seconds | `600` | `600` |

## `CUBEJS_REFRESH_KEY_LOCAL_TIME`

If `true`, interval and cron based [`refresh_key`](/reference/data-modeling/cube#refresh_key)
values are computed from the Cube instance's own clock instead of being fetched with a
`SELECT FLOOR(...) as refresh_key` query. This removes a database round trip per refresh
key check, which for the default 10 second refresh key means one fewer query on nearly
every request.

| Possible Values | Default in Development | Default in Production |
| --------------- | ---------------------- | --------------------- |
| `true`, `false` | `false` | `false` |

Only `every`-based refresh keys are affected. Refresh keys defined with `sql`, and those
using [`incremental`](/reference/data-modeling/pre-aggregations#incremental), still run
against the database because their values cannot be derived from a time interval alone.

<Warning>

Set this variable to the same value on **every** API instance and refresh worker, and keep

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two operational consequences worth adding to this warning:

  1. Flipping the flag invalidates every pre-aggregation once. SELECT FLOOR(...) as refresh_key comes back as a string from several drivers (Postgres bigint/numeric, BigQuery INT64), while evaluateLocalRefreshKey returns a JS number. PreAggregationLoader.contentVersion hashes the invalidation key values, so [{refresh_key: "12345"}] and [{refresh_key: 12345}] produce different content versions and every rollup rebuilds on the first request after the toggle. That is a one-time cost, but on a large deployment it is worth warning about (and it applies in both directions).

  2. Invalidation becomes fleet-synchronized. Today each node caches its refresh-key result independently, so the moment a node notices a new interval is staggered by whenever it last fetched. With synchronized clocks every node flips at the exact same instant, so the primary-query caches across the fleet all miss simultaneously at each boundary. Not incorrect, but a thundering-herd shape that operators should expect.

their clocks synchronized (for example with NTP). Instances whose clocks straddle an
interval boundary compute different refresh keys, which can cause the same
pre-aggregation to be built more than once.

</Warning>

## `CUBEJS_REFRESH_WORKER`

If `true`, this instance of Cube will **only** refresh pre-aggregations.
Expand Down
9 changes: 9 additions & 0 deletions packages/cubejs-backend-shared/src/env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,15 @@ const variables: Record<string, (...args: any) => any> = {
preciseDecimalInCubestore: () => get('CUBEJS_DB_PRECISE_DECIMAL_IN_CUBESTORE')
.default('false')
.asBoolStrict(),
/**
* Evaluates interval and cron based `refreshKey` values from the API instance
* clock instead of issuing `SELECT FLOOR(...) as refresh_key`. Requires clocks
* to be in sync across all API instances and refresh workers, otherwise nodes
* straddling an interval boundary disagree and rebuild pre-aggregations twice.
*/
refreshKeyLocalTime: () => get('CUBEJS_REFRESH_KEY_LOCAL_TIME')
.default('false')
.asBoolStrict(),

/** ****************************************************************
* Common db options *
Expand Down
19 changes: 19 additions & 0 deletions packages/cubejs-backend-shared/test/env.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,25 @@ describe('getEnv', () => {
).toBe(10);
});

test('refreshKeyLocalTime', () => {
delete process.env.CUBEJS_REFRESH_KEY_LOCAL_TIME;
expect(getEnv('refreshKeyLocalTime')).toBe(false);

process.env.CUBEJS_REFRESH_KEY_LOCAL_TIME = 'true';
expect(getEnv('refreshKeyLocalTime')).toBe(true);

process.env.CUBEJS_REFRESH_KEY_LOCAL_TIME = 'false';
expect(getEnv('refreshKeyLocalTime')).toBe(false);
});

test('refreshKeyLocalTime(exception)', () => {
process.env.CUBEJS_REFRESH_KEY_LOCAL_TIME = 'yes';

expect(() => getEnv('refreshKeyLocalTime')).toThrowError();

delete process.env.CUBEJS_REFRESH_KEY_LOCAL_TIME;
});

test('livePreview', () => {
expect(getEnv('livePreview')).toBe(true);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,9 +192,17 @@ export class PreAggregationLoadCache {

public async keyQueryResult(sqlQuery: QueryWithParams, waitForRenew: boolean, priority: number) {
const [query, values, queryOptions] = sqlQuery;
const redisKey = this.queryCache.queryRedisKey([query, values]);

if (!this.queryResults[this.queryCache.queryRedisKey([query, values])]) {
this.queryResults[this.queryCache.queryRedisKey([query, values])] = await this.queryCache.cacheQueryResult(
if (!this.queryResults[redisKey]) {
// A locally evaluated key needs no cache driver or queue, but it must still go
// through this per-request memo. One load reads the invalidation keys several
// times (contentVersion, the returned refreshKeyValues, the refresh queue key);
// re-reading the clock could straddle an interval boundary and have the load
// look a table up under one content version and enqueue it under another.
const local = this.queryCache.localRefreshKeyResult(queryOptions);

this.queryResults[redisKey] = local ?? await this.queryCache.cacheQueryResult(
query,
values,
[query, values],
Expand All @@ -212,9 +220,15 @@ export class PreAggregationLoadCache {
}
);
}
return this.queryResults[this.queryCache.queryRedisKey([query, values])];
return this.queryResults[redisKey];
}

// TODO hashes the whole [query, values, options] tuple, while keyQueryResult stores
// under [query, values]. Every compiler-produced invalidateKeyQueries entry carries an
// options object, so this always returns false, which pins PreAggregationLoader to the
// background-refresh branch whenever waitForRenew is false. Aligning the keys would
// change sync-vs-background refresh semantics for every pre-aggregation, so it needs to
// land on its own rather than riding along with an unrelated change.
Comment on lines +226 to +231

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The analysis here is correct (queryRedisKey on a 3-element tuple never matches the 2-element key keyQueryResult stores under, so notLoadedKey at PreAggregationLoader.ts:134 is always truthy), but this is a pre-existing bug in an unrelated method and a six-line TODO is easy to lose in a diff. Consider filing it as an issue and shortening this to a one-line // TODO(#NNNNN) pointer, so it doesn't read as something this PR introduced or is responsible for.

public hasKeyQueryResult(keyQuery) {
return !!this.queryResults[this.queryCache.queryRedisKey(keyQuery)];
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,12 @@ import { ContinueWaitError } from './ContinueWaitError';
import { LocalCacheDriver } from './LocalCacheDriver';
import { DriverFactory, DriverFactoryByDataSource } from './DriverFactory';
import { LoadPreAggregationResult, PreAggregationDescription } from './PreAggregations';
import { getCacheHash, extractRequestUUID } from './utils';
import {
getCacheHash,
extractRequestUUID,
evaluateLocalRefreshKey,
isValidLocalRefreshKey,
} from './utils';
import { CacheAndQueryDriverType, MetadataOperationType } from './QueryOrchestrator';

export type CacheQueryResultOptions = {
Expand All @@ -44,12 +49,28 @@ export type CacheQueryResultOptions = {
renewCycle?: boolean,
};

/**
* Everything needed to evaluate an `every` based refreshKey without touching a
* database: `FLOOR((utcOffset + unixTimestamp - dayOffset) / interval)`.
*
* `utcOffset` is frozen at compile time, exactly as it already is when baked into
* the emitted SQL string — the orchestrator must never recompute it, or the local
* and SQL paths would stop agreeing across a DST transition.
*/
export type LocalRefreshKeyDescriptor = {
interval: number;
utcOffset: number;
dayOffset: number;
cron?: boolean;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: cron is never read — neither evaluateLocalRefreshKey nor isValidLocalRefreshKey looks at it, and everyRefreshKeyParts only uses it internally in everyRefreshKeySql to pick the branch. It rides along in every serialized invalidateKeyQueries tuple for no consumer. Either drop it from the descriptor (keep it as a local in everyRefreshKeySql) or add a comment saying it is retained deliberately for debuggability.

};

type QueryOptions = {
external?: boolean;
renewalThreshold?: number;
updateWindowSeconds?: number;
renewalThresholdOutsideUpdateWindow?: number;
incremental?: boolean;
localRefreshKey?: LocalRefreshKeyDescriptor;
};

export type QueryWithParams = [
Expand Down Expand Up @@ -144,6 +165,7 @@ export interface QueryCacheOptions {
cacheAndQueueDriver: CacheAndQueryDriverType;
maxInMemoryCacheEntries?: number;
skipExternalCacheAndQueue?: boolean;
localRefreshKey?: boolean;
}

export class QueryCache {
Expand All @@ -155,6 +177,10 @@ export class QueryCache {

protected memoryCache: LRUCache<string, CacheEntry>;

protected readonly localRefreshKeyEnabled: boolean;

protected localRefreshKeyLogged: boolean = false;

public constructor(
protected readonly cachePrefix: string,
protected readonly driverFactory: DriverFactoryByDataSource,
Expand All @@ -181,6 +207,31 @@ export class QueryCache {
this.memoryCache = new LRUCache<string, CacheEntry>({
max: options.maxInMemoryCacheEntries || 10000
});

this.localRefreshKeyEnabled = options.localRefreshKey ?? getEnv('refreshKeyLocalTime');
}

/**
* Evaluates an interval based refresh key locally, or returns null when the SQL
* path must be used: the feature is off, the key is one we cannot reproduce
* (`refreshKey.sql`, incremental), or the descriptor is malformed.
*/
public localRefreshKeyResult(queryOptions?: QueryOptions): [{ refresh_key: number }] | null {
if (!this.localRefreshKeyEnabled || !isValidLocalRefreshKey(queryOptions?.localRefreshKey)) {
return null;
}

// Logged on first use rather than from the constructor, where subclass field
// initialization has not run yet and `this.logger` may not be usable.
Comment on lines +224 to +225

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the stated reason isn't right. logger is a TypeScript parameter property, so it is assigned before the constructor body runs — this.logger(...) from the constructor would work. The actual (and better) justification for lazy logging is that the warning should only fire when a descriptor is genuinely used, not on every QueryCache construction in a multi-tenant process. Worth rewording so the comment doesn't send the next reader looking for an initialization-order hazard that isn't there.

if (!this.localRefreshKeyLogged) {
this.localRefreshKeyLogged = true;
this.logger('Local refresh key evaluation enabled', {
warning: 'Interval based refresh keys are evaluated from this instance clock. ' +
'Clocks must be in sync across all API instances and refresh workers.',
});
}

return evaluateLocalRefreshKey(<LocalRefreshKeyDescriptor>queryOptions?.localRefreshKey);
}

public getCacheDriver(): CacheDriverInterface {
Expand Down Expand Up @@ -864,6 +915,14 @@ export class QueryCache {
public async loadRefreshKey(q: QueryWithParams, expireSecs: number, options: LoadRefreshKeyOptions) {
const [query, values, queryOptions] = q;

// A locally evaluated key is free, so there is nothing to cache and no queue to
// wait on. The value is quantized to the interval, so repeated calls agree
// except across a boundary — the same window the cached SQL result had.
const local = this.localRefreshKeyResult(queryOptions);
if (local) {
return local;
}

Comment on lines +918 to +925

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refreshKeyRenewalThreshold is silently bypassed on the local path.

The SQL path below resolves renewalThreshold: this.options.refreshKeyRenewalThreshold || queryOptions?.renewalThreshold || 2 * 60, and PreAggregationLoadCache.keyQueryResult does the same (PreAggregationLoadCache.ts:211). Because the cached refresh-key value is only re-read every renewalThreshold seconds, the observed invalidation cadence today is effectively max(interval, renewalThreshold). The local path returns before any of that, so the cadence becomes exactly interval.

For the defaults this is a no-op (refreshKeyRenewalThresholdForInterval returns min(interval/10, 300), always below the interval). But orchestratorOptions.queryCacheOptions.refreshKeyRenewalThreshold is a supported user knob (optionsValidate.ts:128), and someone who set it to e.g. 300 against the default every: '10 seconds' key is deliberately throttling invalidation 30×. Turning this flag on would silently restore 10s invalidation of their query cache and pre-aggregation content versions — i.e. a large increase in real DB queries, which is the opposite of the PR's intent.

Cheapest fix is to quantize to the effective threshold rather than the raw interval, e.g. pass Math.max(descriptor.interval, this.options.refreshKeyRenewalThreshold || queryOptions?.renewalThreshold || 0) into evaluateLocalRefreshKey. At minimum this interaction should be called out in the docs.

Fix this →

return this.cacheQueryResult(
query,
values,
Expand Down
31 changes: 30 additions & 1 deletion packages/cubejs-query-orchestrator/src/orchestrator/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import crypto from 'crypto';

import { getProcessUid } from '@cubejs-backend/shared';
import { QueryKey, QueryKeyHash } from '@cubejs-backend/base-driver';
import { CacheKey } from './QueryCache';
import { CacheKey, LocalRefreshKeyDescriptor } from './QueryCache';

/**
* Unique process ID regexp.
Expand Down Expand Up @@ -32,6 +32,35 @@ export function getCacheHash(queryKey: QueryKey | CacheKey, processUid?: string)
}
}

/**
* Evaluates an `every` based refreshKey from the local clock, producing the same
* row shape the equivalent `SELECT FLOOR(...) as refresh_key` would return.
*
* `nowMs` is not floored to whole seconds first: for integer `x`, fractional
* `f` in [0, 1) and `interval >= 1`, floor((x + f) / interval) === floor(x / interval),
* which is also why the fractional seconds of `EXTRACT(EPOCH FROM NOW())` never
* mattered on the SQL path.
*/
export function evaluateLocalRefreshKey(
descriptor: LocalRefreshKeyDescriptor,
nowMs: number = Date.now(),
): [{ refresh_key: number }] {
const { utcOffset, interval, dayOffset } = descriptor;

return [{ refresh_key: Math.floor((utcOffset + nowMs / 1000 - dayOffset) / interval) }];
}

/**
* A malformed descriptor must fall back to the SQL path rather than produce a
* garbage refresh key, which would silently invalidate everything downstream.
*/
export function isValidLocalRefreshKey(descriptor?: LocalRefreshKeyDescriptor): boolean {
return !!descriptor &&
Number.isFinite(descriptor.interval) && descriptor.interval > 0 &&
Number.isFinite(descriptor.utcOffset) &&
Number.isFinite(descriptor.dayOffset);
}

/**
* Extracts the UUID prefix from a request ID by stripping the `-span-N` suffix.
*/
Expand Down
104 changes: 104 additions & 0 deletions packages/cubejs-query-orchestrator/test/unit/PreAggregations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,110 @@ describe('PreAggregations', () => {
});
});

describe('local refresh key', () => {
const REFRESH_KEY_SQL = 'SELECT FLOOR((UNIX_TIMESTAMP()) / 600) as refresh_key';
const descriptor = { interval: 600, utcOffset: 0, dayOffset: 0, cron: false };

const newLoadCache = (localRefreshKey?: boolean) => {
const cache = new QueryCache(
'TEST',
mockDriverFactory as any,
// eslint-disable-next-line @typescript-eslint/no-empty-function
() => {},
{
cacheAndQueueDriver: 'memory',
localRefreshKey,
queueOptions: async () => ({ executionTimeout: 1, concurrency: 2 }),
},
);
(cache.getCacheDriver() as LocalCacheDriver).reset();

const preAggregations = new PreAggregations(
'TEST',
mockDriverFactory as any,
// eslint-disable-next-line @typescript-eslint/no-empty-function
() => {},
cache,
{ queueOptions: async () => ({ executionTimeout: 1, concurrency: 2 }) },
);

return new PreAggregationLoadCache(
mockDriverFactory as any,
cache,
preAggregations,
{ dataSource: 'default' },
);
};

test('keyQueryResult evaluates locally without querying the datasource', async () => {
const loadCache = newLoadCache(true);

const result = await loadCache.keyQueryResult(
[REFRESH_KEY_SQL, [], { external: true, renewalThreshold: 60, localRefreshKey: descriptor }],
false,
10,
);

expect(result).toEqual([{ refresh_key: Math.floor(Date.now() / 1000 / descriptor.interval) }]);
expect(mockDriver!.executedQueries).toEqual([]);
});

test('keyQueryResult still queries when the flag is off', async () => {
const loadCache = newLoadCache(false);

await loadCache.keyQueryResult(
[REFRESH_KEY_SQL, [], { external: false, renewalThreshold: 60, localRefreshKey: descriptor }],
false,
10,
);

expect(mockDriver!.executedQueries).toEqual([REFRESH_KEY_SQL]);
});

test('keyQueryResult still queries an incremental key that carries no descriptor', async () => {
const loadCache = newLoadCache(true);
const incrementalSql = 'SELECT CASE WHEN NOW() < $1 THEN FLOOR((UNIX_TIMESTAMP()) / 3600) END as refresh_key';

await loadCache.keyQueryResult(
[incrementalSql, [], {
external: false,
renewalThreshold: 300,
incremental: true,
renewalThresholdOutsideUpdateWindow: 86400,
}],
false,
10,
);

expect(mockDriver!.executedQueries).toEqual([incrementalSql]);
});

// The per-request memo must pin the value for the life of one load. A single
// load reads the invalidation keys several times (contentVersion, the returned
// refreshKeyValues, the refresh queue key); if the clock were re-read, a load
// crossing an interval boundary would look a table up under one content version
// and enqueue it under another.
test('keyQueryResult is stable across an interval boundary within one load cache', async () => {
const loadCache = newLoadCache(true);
const key: [string, any[], Record<string, any>] =
[REFRESH_KEY_SQL, [], { external: true, renewalThreshold: 60, localRefreshKey: descriptor }];

const nowSpy = jest.spyOn(Date, 'now').mockReturnValue(600_000);
try {
const first = await loadCache.keyQueryResult(key, false, 10);
expect(first).toEqual([{ refresh_key: 1 }]);

// Advance well past the next boundary.
nowSpy.mockReturnValue(1_200_000);
const second = await loadCache.keyQueryResult(key, false, 10);

expect(second).toEqual(first);
} finally {
nowSpy.mockRestore();
}
});
});

describe('loadAllPreAggregationsIfNeeded', () => {
let preAggregations: PreAggregations | null = null;

Expand Down
Loading
Loading