feat(app): DQX Studio dqlake integration - #1428
Open
berrybluecode wants to merge 113 commits into
Open
Conversation
…esults UI Brings the DQX × dqlake Rules Registry work into dqx-dqlake-integration: reusable/versioned/governed registry rules, monitored-table bindings, Table Spaces (data products), the dqlake-ported results UI, the Lakebase OLTP backend, and the supporting services + migrations. Co-authored-by: Isaac
~110 UI/UX tweaks, bug fixes and features layered on rules-registry: schedule scope, admin-settings restructure (Governance/Tags/Danger), results score animation + 2D chart zoom/pan, Genie change-awareness, profiler suggestions on the profile page (create-on-apply), runs-history timezone/type/duration fixes, steward sort defaults, review-status rename/recolor, Compute OBO + clusters scope, and more. See docs/superpowers/plans/2026-07-11-ui-tweaks*. Co-authored-by: Isaac
Drop the -ml-3 gutter offset on the per-message "GENIE" header so the mark + label line up with the response prose/SQL/actions below. Co-authored-by: Isaac
The failed-to-load AlertCircle used h-10 w-10 (40px = fractional 5/3 of lucide's 24px grid → soft/blurry strokes). Bump to h-12 w-12 (integer 2×, crisp), matching the empty-state Boxes/Library icons. Co-authored-by: Isaac
…2-112) The new-rule create page rendered its own <h1> above the form and passed no headerTitle, so the page-variant header rendered an empty left <div/> with Save/Submit floated right via justify-between. Move the title into the form's headerTitle prop and drop the standalone heading so the actions sit inline with the title, matching the existing-rule detail page and the Monitored Table / Table Space headers. Co-authored-by: Isaac
The admin "Reset Database" full-reset DELETEs all app-owned dq_* data, including the fresh-install DEFAULT content in dq_app_settings (run review statuses and the reserved dimension/severity label definitions). Previously those were only re-seeded lazily at the next app startup, so after a reset the tables sat empty until a restart. DatabaseResetService now re-provisions those defaults in the same request, immediately after the DELETE phase, by re-running the app's own first-boot seed routines (AppSettingsService.seed_*_if_absent) over the same OLTP executor the deletes ran on — no seed data duplicated. A full reset therefore lands on a clean fresh-install state, not an empty one. Re-provisioning is best-effort (failures recorded under seed:<name> in failed_tables, never fatal) and the existing safety scoping is unchanged: dq_migrations and admin role mappings are still preserved. Co-authored-by: Isaac
Databricks Apps rejects `clusters` as a user_api_scope with 400 INVALID_PARAMETER_VALUE, which failed the entire app-resource update (cascading to grants/roles) so bundle deploy never reached the app. There is no valid Apps scope for cluster listing; the Compute picker degrades to [] gracefully (B2-105). Co-authored-by: Isaac
…-114) The standalone "As JSON" button on the Implementation tab of the new registry-rule create page is relocated into a ⋮ (MoreVertical) overflow menu in the page header, mirroring the existing-rule detail page. - registry-rules.new.tsx: adds a shadcn DropdownMenu ⋮ trigger wired via the form's `headerActions` prop, with an "As JSON" DropdownMenuItem that drives the form's controlled `jsonDialogOpen`/`onJsonDialogOpenChange`. - RegistryRuleFormDialog.tsx: removes the inline "As JSON" SectionHeader button (and now-unused Braces import). Reuses existing i18n keys (actionAsJson, actionsMenuLabel). Behavior of the JSON view/edit dialog is unchanged. Existing-rule detail page header is untouched. Co-authored-by: Isaac
Once the FadeIn entrance settles, motion leaves a transform/will-change on the element, keeping it on a GPU-composited layer that rasterizes SVG icons soft/blurry (text stays crisp). Clearing both on animation-complete drops the layer so at-rest content is pixel-crisp — fixes the Table Spaces empty-state Boxes icon (B2-111 follow-up). Co-authored-by: Isaac
- B2-115: add-rules picker preselects + disables already-applied rules - B2-116: apply-rules staged row reflects 'follow latest by default' setting - B2-117: fix blurry Table Spaces empty-state icon (Boxes glyph -> LayoutGrid) - B2-118: require a FRESH draft run since last edit before submit - B2-120: remove 'Steward:' line from monitored-table title bar - B2-121: Danger Zone dark-mode contrast - B2-122: seed 'inherit to child objects' toggle default from admin setting - B2-123: remove dead 'Top values not available' profiling panel - B2-124: hide dimension tag on profiler suggested checks - B2-125: rename 'Filter columns' -> 'Search columns' - B2-126: correct validation-run duration (job wall-clock span) - B2-127: auto-surface creator owner-equivalent full-permission grant - B2-128: homepage overview chart counts only full runs vs approved tables - B2-129: profiling-suggested rules attributed 'Co-authored with AI' - B2-130: dedup already-applied rule/column combos in AI Suggest (case-normalized) Co-authored-by: Isaac
Drop the inline uppercase badge (B2-115); the disabled/checked row state already conveys already-applied rules. Kept the aria-label for screen readers. Co-authored-by: Isaac
…132..B2-140) - B2-131: remove visible 'Already applied' badge from add-rule picker - B2-132: 'Build with AI' generates low-code rules first (low-code->dqx_native->sql), with a Python port of the low-code AST->SQL compiler - B2-134: reword stale-draft submit block -> 'You cannot submit for approval without first running your draft' - B2-135: drop the draft-runs explainer banner; auto-default the run-mode picker to Published+Draft when the latest run is a draft - B2-136: mark draft-run points with a 'Draft' label in the score-trend tooltip - B2-137: reword needs-draft hint -> 'A draft run is required before you submit for review' - B2-138: reactivate loading spinners when switching run mode (key off isFetching) - B2-139: render the dot for a table's first/only score-trend point - B2-140: honor an explicit rule-type request in the AI prompt (straight to that generator; fail rather than silently switch modes) Co-authored-by: Isaac
…ff (B2-141/B2-142) - B2-141: the Monitored Tables list computed draft check-counts with a per-never-approved-binding render loop (~3N + 2*sum(R) sequential OLTP round-trips). Batched to 3 constant grouped queries (list_applied_rules_many, get_rules_many, get_versions_many) with count parity preserved. - B2-142: hide the Review & Approve sidebar item and its trailing divider when the app-wide approvals mode is 'disabled' (no review queue). Co-authored-by: Isaac
…-on-tag)
Squash-merge of dqx/apply-on-tag. Lets a registry rule's {{slot}}
placeholders bind to governed Unity Catalog column tags, so a published rule
fans out to every monitored table whose columns carry matching tags.
- Rule authoring: per-slot governed-tag chips in the Registry "Columns used"
panel (any governed tag via SHOW GOVERNED TAGS; app-native description
tooltips). Stored in the rule's user_metadata (slot_tags).
- Matching: any-overlap within a slot + family/type compatibility always
enforced; full Cartesian product across slots.
- Admin toggle (tag_auto_apply, default off): ON auto-attaches matches;
OFF surfaces them in the existing "Suggest rules" dialog ("Matched tag <x>").
- Auto-apply runs OBO (register / bulk-register / open) so it reads the
caller's tags; the app SP has no user-catalog grant. Background SP sweep
as a best-effort backstop. Add-only + origin marker + removal tombstone
so a user-removed auto rule is never resurrected.
- About-tab schema table gains a Tags column (overflow → tooltip).
Co-authored-by: Isaac
Squash-merge of dqx/demo. Two bodies of work. DEMO CONTENT — an ADMIN-only "Deploy demo content" action (+ committed CLI wrapper) that seeds a workspace with a realistic DQX Studio deployment: 15 reusable rules across all three authoring modes and every quality dimension, mapped onto 5 sample tables and 2 data products, with a dqlake-faithful 9-week quality story built from REAL engine runs then re-dated into the past (incident dip+recover, steady improvement, a rule added mid-history, a rule tightened). Governed class.* column tags on a few rules showcase tag-based auto-apply. Runs ~30min on a background daemon thread; status is polled and self-heals a stale run. Explicit-approval invariant preserved throughout. Deployed and verified end-to-end: no orphan/latent trend points on any chart (overall, dimension, severity, rule, product), overview/homepage headline == trend's final point, and per-table history is a varying 9-point series. New: backend/demo/ (manifest, datagen, redate, seed_service, status), the demo deploy/status admin routes + DI wiring, the Deploy-demo-content admin card (4-locale i18n), the dqx_studio_demo SP-owned bundle schema, and scripts/ seed_demo.py. Core scoring/materialisation/runner logic untouched. RULE-NAME DISPLAY + COLUMN ATTRIBUTION (core, benefits all users) — - Multi-column rules: each per-column rendered check gets a distinct name (suffixed) so per-column results AND failure counts stay separate, while Results by-rule and Genie DISPLAY the underlying rule name from user_metadata['name'] (grouping still by registry_rule_id; Genie keeps its COUNT(DISTINCT check_name)/col_set rollup + a fan-out instruction). - sql_query/low-code checks no longer pass an unsupported `columns` argument (DQX's validator rejected it); mapped columns ride in user_metadata and the attribution view recovers them via COALESCE, so the by-column breakdown populates for every mode. sql_expression keeps its declared columns arg. - Failed-records highlighter shows the generic rule name and recovers the failing column(s) from the metadata carrier when the struct's columns is empty. - Results tab no longer shows a stale run after completion (invalidate after the score-cache recompute lands, not before). Full suite green: 1128+ backend tests, 395 UI tests, basedpyright 0 errors, ruff clean. Design specs + implementation plans under docs/superpowers/. Co-authored-by: Isaac
The suggest-rules feature retrieves from the dq_rule_embeddings corpus, which the HTTP approve route populates via RuleEmbeddingsService.embed_and_store. The seeder drives RegistryService.approve directly (no route), so demo rules were never embedded — they never surfaced as suggestions, unlike hand-authored rules (which go through the route). That's the reported gap. DemoSeedService now takes an optional RuleEmbeddingsService and embeds each rule in _build_rules after it reaches approved — both the freshly-created and the reused-from-a-prior-seed branches (a reused rule may never have been embedded). Wired through get_demo_seed_service (FastAPI) and the seed_demo CLI. embed_and_store is best-effort (no-op when no embedding endpoint is configured, never raises), and _embed_rule additionally guards any escape, so this never aborts the ~30min seed. Tests: created rules embedded, reused rules embedded, embed-failure survived, and no-embeddings-service is a no-op. 400 related tests pass; ruff clean. Co-authored-by: Isaac
… import - Add Export to YAML (DQX check-list or ODCS data contract) on the Rule Registry, Monitored Tables, and Table Spaces (page-level + per-row + detail), backed by a new /api/v1/export service and routes. - Add DQX-YAML and ODCS bulk contract import workspaces + import-registry-rules parsing/normalization helpers and routes. - Move rule filter (WHERE) and pass threshold to be per-applied-rule (row_filter + pass_threshold on dq_applied_rules); render row_filter into the DQX check filter and carry pass_threshold in user_metadata. - Add pending-application store + activation so bulk-imported rules link on approval. - Relocate table-level comments to the About tab and add per-run comments to the Results tab. - Migrations, models, services, i18n (en/es/it/pt-BR), and tests updated.
…x into dqx-dqlake-integration # Conflicts: # app/src/databricks_labs_dqx_app/backend/dependencies.py # app/src/databricks_labs_dqx_app/backend/migrations/__init__.py # app/src/databricks_labs_dqx_app/backend/migrations/postgres.py # app/src/databricks_labs_dqx_app/backend/models.py # app/src/databricks_labs_dqx_app/backend/routes/v1/registry_rules.py # app/src/databricks_labs_dqx_app/backend/services/apply_rules_service.py # app/src/databricks_labs_dqx_app/backend/services/materializer.py # app/src/databricks_labs_dqx_app/ui/lib/registry-rule-conversion.test.ts # app/src/databricks_labs_dqx_app/ui/routes/_sidebar/monitored-tables.$bindingId.tsx # app/tests/test_monitored_tables_routes.py # app/tests/test_registry_rules_routes.py
…param limit)
Monitored tables with many applied rules produce a checks list that overflows
the Jobs run_now job_parameters 10,000-char cap. Add run_config_store to write
the full config to {DQX_WHEELS_VOLUME}/run-configs/{run_id}.json and pass a tiny
{"__staged__": "<path>"} stub instead; the task runner resolves the stub back to
the full config. Wire it through binding_run_service / job_service / runner, add
the per-rule run-config UI (RuleConfigCard + config page), and update i18n.
Governed class.* column tags (class.location, class.credit_card) need ASSIGN on the tag policy — the app SP typically lacks it, so tags were never applied to the demo tables (best-effort skip). The admin who clicks "Deploy demo content" usually DOES hold ASSIGN, and their OBO token is available in the deploy request. DemoSeedService now takes an optional tagging WorkspaceClient (settable via set_tagging_ws); the deploy route hands it the caller's obo_ws before launching the seed thread. _assign_column_tag prefers that OBO client, falling back to the SP (CLI / tests / no-OBO). Tagging is the seed's first phase, so the OBO token is still fresh on the background thread. Still fully best-effort — a caller without ASSIGN degrades to a logged skip, never aborting the seed. No SP tag-read grant is needed: tag-based auto-apply already reads column tags via the calling user's OBO (an SP read of information_schema.column_tags returns nothing by design), and `account users` already has USE_SCHEMA+SELECT on the demo schema via the bundle — so admin-OBO assignment + the existing grant make the whole showcase self-contained. Tests: OBO client preferred when set (SP untouched); SP fallback when no OBO; existing best-effort-swallow + SDK-shape tests still hold. 48 pass, ruff clean. Co-authored-by: Isaac
The prior OBO approach used the UC entity-tag-assignments API, which needs the coarse `unity-catalog` OBO scope the app doesn't mint (it uses granular catalog.*:read). Rather than widen every user's OBO token, assign governed class.* tags with `SET TAG ON COLUMN <fqn>.<col> `<dotted.key>`` DDL — the governed key backtick-quoted so its dot is literal (verified live: this DDL applies class.location to a column; the older ALTER COLUMN SET TAGS form rejects the dot, SET TAG does not). SET TAG needs only the `sql` warehouse scope, which the app already has. The DDL runs through the caller's OBO SqlExecutor (set_tagging_sql, supplied by the deploy route from get_obo_sql_executor) so it executes as the deploying admin who holds ASSIGN on the tag policy, falling back to the SP-owned demo executor for CLI/tests/no-OBO. Still fully best-effort — a caller without ASSIGN degrades to a logged skip, never aborting the seed. Replaces the entity_tag_assignments API call and the tagging_ws plumbing. Tests updated to assert the SET TAG DDL (dotted key backtick-quoted), OBO-sql preferred when set, SP fallback, and best-effort error swallow. 49 pass; basedpyright + ruff clean. Co-authored-by: Isaac
Demo showcase: add two governed-tag rules that are created+approved but NOT
applied to any binding, so the tag-based suggest/auto-apply flow has a live
match to demonstrate:
- iso2_country "Valid ISO 3166-1 alpha-2 country code" (regex ^[A-Z]{2}$),
slot tag class.location — matches customers.country_code
- card_not_null "Card last-four is present" (is_not_null),
slot tag class.credit_card — matches payments.card_last4
Apply Rules UI: surface the matched governed tag(s) so the match is legible.
- New pure helper computeMatchedTagsForSlot(slotTags, columnTags, slot, column)
= intersection of a slot's suggested governed tags (rule user_metadata.slot_tags)
and the tags actually applied to the mapped column (useGetTableTags), order
preserved (+ 6 unit tests).
- By-rule lens (MappingChips via RuleConfigCard): small muted tag chips under
each mapping row when the slot's suggested tag matches the column's applied tag.
- By-column lens (RulesByColumn): matched tag chip(s) inline after the rule name.
Chips reuse the ColumnTagsCell badge style. No backend change — slot_tags is
already on user_metadata and column tags come from the existing table-tags query.
app-check clean (0 type errors); 401 UI tests pass (395 + 6); 57 demo tests pass.
Co-authored-by: Isaac
…de it The matched governed-tag chip rendered to the RIGHT of the column chip; move it to the line BELOW so it reads as metadata about the column above it. The column-chip wrapper is now a flex column (chip on top, tag chips underneath), and the mapping row's chip container switches to items-start so the stacked tag line doesn't vertically-center the sibling chips / the "+ Apply to another column" button. app-check clean; 401 UI tests pass. Co-authored-by: Isaac
Full redesign of the DQX Studio admin settings page (route /config → /settings):
restructured tabs, sentence-case titles, consistent card spacing, universal
auto-save (no Save buttons / dividers), and per-tab A-Z card ordering.
- Tabs: General, then AI/Compute/Entitlements/Governance/Tags (alphabetical),
then Danger zone; General uses a sliders icon; AI tab uses the AI gradient.
- General: Display timezone + Global results tab (trimmed copy).
- AI: renamed from "AI settings"; "Manage in AI Gateway" link; auto-save;
purple inner-card borders + fuchsia toggle; endpoint disabled when AI off.
- Governance: Run review statuses (restyled like Rule labels, hex color
picker via row swatch, resilient single-default); Rules Registry rename +
state-dependent hints; Approvals workflow with "Bypass approvals for
automatic rule upgrades" (gated on mode); Draft runs ("Draft run sample
size", trimmed copy); Data retention (Invalid results + All other data as
two mini cards with number + days/months/years, defaults 1mo/3mo).
- Tags: split Built-in / Custom; Key + Description + delete on one row;
"Allow custom values" inline-left of "Add value"; trimmed hints/warnings.
- Entitlements: backend endpoint surfacing workspace admins + app owners as
disabled rows; DQ-steward principal picker; user-level entitlements now
resolve (identity folded into role resolution); name-only entitlement
dropdown; add-on-select; dynamic per-OS hard-refresh hint; de-jargoned
toast; Permissions card renamed, "Cascade permissions by default" defaults on.
- Compute: fixed picker load race (suspense hooks), dqlake-correct options,
reused Random-sample control for Draft runs, auto-save.
- Danger zone: shortened warning to "This cannot be undone."; button "Reset".
Co-authored-by: Isaac
Squashed rule-editor work (granular history preserved on dqx/rule-creation-improvements). Rule editor / authoring flow - Rule-type consolidation, merged condition selector with search + richer animation, entry-flow polish, cursor tooltips, grouped/auto-family operators - Unified condition row; native two-cell layout; keep-at-least-one-condition - Columns-used gating, animated column cards, bulleted save-gate tooltip - About-tab spacing + Implementation headers matched to About/Permissions - Warn + clear conditions when a column is retyped to an incompatible family Filters, joins & group-by - Row filter as a low-code builder for native/low-code (persists filter_ast, round-trips); SQL mode uses the code editor (auto-height, grows with content) - Joins authored inline in SQL mode; single-table join pick modal; join-key input-side column list = declared slots only - Dropped redundant helper texts Reference checks (foreign_key, has_valid_schema) - ref_table uses the single-table pick modal, shown before ref_columns and marked mandatory (the Studio has no in-memory-DataFrame path) Condition Builder operator catalog (DQ-steward review) - Length, text-format (uuid/ipv4/regex/case/digits), numeric and temporal predicates; Luhn via the built-in luhn_check() - AI sentiment operators (ai_analyze_sentiment) styled purple with an icon - All compiled SQL passes is_sql_query_safe() - Backend low-code compiler mirrored so the AI generator can propose them Overview & detail surfaces - Rules overview: Export moved into the selection action bar (exports ticked rows via a new rule_id[] filter); deprecated/rejected rules selectable for bulk export/delete; overview Export button removed - Tables & Spaces overview: per-item Export moved from the page header into each row's action - Tables & Spaces detail: Export folded into the existing ⋮ actions menu - AI generator: friendly rule-type names + prefer close-match basic checks; fix Build-with-AI on a brand-new rule - Settings: governed-key import; tag-picker drilldown Co-authored-by: Isaac
Co-authored-by: Isaac
…l/overview Resolves the two open items from the rule-creation session. Overview Export (Tables & Spaces): moved off the per-row action into the selection action bar, exporting exactly the ticked rows — mirroring the Rules overview. Adds binding_id[] / product_id[] filters to exportMonitoredTables / exportDataProducts (route + ExportService, post-filtering the listed summaries/products). Revert-to-draft: new pending_approval -> draft endpoints for monitored tables (revertMonitoredTable, walks each pending check back to draft then flips the binding) and data products (revertDataProduct, DataProductService.revert). Gated to authors-and-above — the submit counterpart, distinct from an approver's reject (no rejected audit trail). Surfaced as an Undo2 action on the detail pending banner / space header AND the overview rows. View changes: wired MonitoredTableDiffDialog / TableSpaceDiffDialog into the table-detail pending banner and the space-detail header (the overview rows already had it), so the read-only diff is available everywhere a submission is pending. ACTIONS_COL_WIDTH comment updated for the new icon set (Export out, Revert in). i18n: revertAction / toastReverted / toastRevertFailed across en/pt-BR/it/es. Tests: export id-filter selection, revert service + route (incl. 409/404/RBAC). Co-authored-by: Isaac
…-rework) Squash-merge of dqx/rules-threshold-rework. Reimplements the per-rule/per-column "pass threshold" as a warn-when-<X%-of-tests-pass gate, with a resolution chain (per-column → per-rule → registry-rule default → admin default) and breach indicators throughout the results UI. Highlights: - Threshold rework: admin default in its own "Pass thresholds" settings card; per-rule threshold in Implementation→Advanced; per-column overrides via the ThresholdPill (Mixed only with 2+ divergent columns; single-column collapses to the column value; no "*" when equal to default; fixed-width marker slot). - Threshold freeze: the materializer stamps the resolved effective threshold into each check's user_metadata, AND the version snapshot now persists/ restores per-rule pass_threshold so approved-version runs freeze the intended value (0% included) instead of falling back to the admin default. - Breach surfacing: BreachIcon (higher-contrast, dark-mode-aware) on drilldown rows (by dimension/severity/rule/column), on the Overall score, at each breaching point on the score-over-time chart, and in the failing-records cell hover. Tooltip copy: "Pass rate below threshold". - Bulk export dialog (rules + tables) + revert-to-draft / view-changes on detail and overview. - AI-suggest: embed slot family/cardinality + check function into the rule embed text; per-column retrieval; top-K=20; same-column multi-slot guard; judge prompt favouring universal integrity checks (profiling-stats + earlier over-eager prompt reverted). Co-authored-by: Isaac
Squash-merge of dqx/bug-bash-v4 (Mini Bug Bash feedback doc). Highlights: - Apply Rules: single-open by-rule accordion, SQL logic display, threshold clamp + dirty-detection, by-column column-aware selectability, 3-dot menu. - Overview pages: overlay bulk action bar (no table shift), export icon, bulk Reject on rules, skip-modal single-format export, consistent headers. - Results: v1 version markers + demo version-freeze back-dating, comments modal inline with review status, double-click zoom reset, always-amber breach icon, score-card icon alignment. - Restored Drafts & Review screen. Demo runs profiling + seeds an unmapped SSN rule pending approval. Rule diffs render full embedded check JSON. Table-space # Checks computed at read time. Genie panel wider + send icon. - Settings: bypass-approvals default true, threshold copy, approvals-mode description, pass-threshold input sizing, rules results-tab admin toggle, global results dropdown filters. - Rule authoring: generate-test-data full-width, searchable row-filter picker, SQL rule-type shortcuts in Basic Checks, JOIN example in SQL placeholder. Permissions: bin icon for revoke, revocable owner privileges (with principal_id-only enforcement matching — security), Spaces column order aligned with Tables. Lakebase max CU raised to 4. Deferred: Genie space re-implementation (item 19, post-reseed); column-ref in condition entry boxes (item 42, needs design). Live-app diagnosis pending: suggestion-engine latency (15), AI model setting verify (30), entitlements render check (32), compute picker error (38). Co-authored-by: Isaac <isaac@example.com>
Second squash from dqx/bug-bash-v4 after the initial 29219ad merge: - Item 52: reinstate by-column data-type compatibility filtering in Apply Rules → By Column (only rules with a slot family compatible with the target column are selectable). - Item 55: SQL predicates containing a JOIN (or full SELECT) now compile to a runnable sql_query check instead of an unrunnable sql_expression. - Item 53: bold rule/check/test in the Apply Rules glossary tooltip. - Item 54: Generate test data button sized to match Run test (not full-width). - Carries the item 39/40 revisions (SQL shortcuts inside Basic Checks; joins box removed in favour of a JOIN example in the SQL predicate placeholder). Co-authored-by: Isaac <isaac@example.com>
vb-dbrks
requested changes
Aug 14, 2026
vb-dbrks
left a comment
vb-dbrks
left a comment
Contributor
There was a problem hiding this comment.
had a look at the AI/retrieval path rather than the whole PR - the rules registry suggester specifically. nothing here contradicts the approval, it's mostly perf plus one docstring thing.
main one is the per-column fan-out in the suggester - each column triggers its own embedding call and its own full read+parse of dq_rule_embeddings, so cost scales with table width inside a single request. that's independent of the vector search discussion and imo the biggest win available here.
rest is smaller - the dual write to the VS index wants removing alongside the provisioner, and the SP vs OBO rationale in dependencies.py contradicts what AIGateway documents for itself.
vb-dbrks
reviewed
Aug 14, 2026
Drop the unused vector store path in favor of AI bootstrap, lock jobs compute to serverless in the UI, and clean up settings/schema flows (auto-assign copy, notes removal) from the bugbash decisions.
Remove the mandatory-tag button, badges, and helpers from AI Rule Suggestions so authors only add the selected mappings.
ghanse
reviewed
Aug 14, 2026
Collaborator
There was a problem hiding this comment.
This looks good overall, a few findings below:
SQL Parameterization and Escaping
Parameterized SQL statements are built using Python f-strings and called through _sql.execute(...) and _sql.query(...)`. Any future safety mechanisms we introduce will require updates across every f-string.
We could add a small query-builder to handle the common CRUD statements. For example:
# current:
self._sql.execute(
f"UPDATE {self._table} SET pinned_version = {value} WHERE id = '{e}'"
)
# suggested:
self._sql.update( # builds and executes the UPDATE statement
self._table, set={"pinned_version": value}, where={"id": rule_id}
)
Code Quality
A few minor code quality nits. Verify if these need fixing:
- There are several very large files with >1,200 lines of code
- There are several instances of linter silencing
- There are a lot of
except Exceptionblocks that only log
Fix Run now ignoring sample size on approved runs, and let schedules persist schedule_sample_size (default remains full table).
The existing suggester tests cover the plumbing well, but they replace both the retriever and the judge with fakes, so nothing in the suite can tell us whether the suggestions are any good. That is why the recent retrieval work (batched embeddings, corpus caching, the new MAX_RETRIEVAL_COLUMNS cap) could land fully green without anyone being able to say whether quality moved. This adds the missing half: a labelled fixture set plus a scorer. Tier 1 lands here, running in make app-test in about three seconds with no workspace and no tokens. The pipeline under test is real, including the suggester, the cosine retriever, the embeddings service, the AI gateway and every post-process gate. The only faked surface is the outbound serving_endpoints.query call, replayed from recorded fixtures. Replaying at the HTTP boundary rather than at the retriever or gateway seam is deliberate: a stub at the seam can pass while the real path is inert, which is exactly how the MCP telemetry wrapper stayed green while sending nothing. Scoring reuses compute_mapping_hash as the identity for a suggestion, so a label and a prediction compare exactly and a mapping written in a different key order is not counted as a different answer. Alongside precision, recall and precision@k there is a separate retrieval recall, which isolates the rule the judge never saw from the rule it saw and passed over, and a must-not-suggest list, which is the only way to catch a rule bound to a plausible-but-wrong column since every structural gate accepts it. The harness also has to fail when quality regresses, or it is just a second copy of the happy path. Four sensitivity tests inject a different regression each by varying the input data rather than patching internals, and assert the matching metric moves. Baseline on the current head is precision 0.824, recall 1.000, three planted violations detected. What Tier 1 does not cover is stated plainly in the module docstring: the judge is held fixed as a recorded oracle, so a change to its prompt is invisible here and has to be measured live. The live tier is next.
The harness observes the outbound call count at the serving-endpoint boundary, which is the only place the batching is visible. Before the batch fix a seven-column table cost seven embedding round trips; a regression there gives the same answers, just slower, so no correctness test would catch it.
…t misses by stage The 15-rule corpus turned out to make the eval unable to answer the questions it was built for. With top-20 per column and a union cap of top_k*3, a corpus that small hands the judge every rule, so retrieval was doing nothing and neither the per-column fan-out nor the 64-column cap could be measured at all. The corpus is now 82 rules and retrieval is genuinely selective, offering between 33 and 60 of them depending on the table. Sixty-seven of the new rules are distractors: plausible checks across healthcare, finance, supply chain, geo, telemetry and HR whose only job is to give retrieval something to rank against. They can never become a prediction, because the oracle never names them, so they shift ranking and nothing else. Five tables join the three that were there. The wide one is 90 columns with its five distinctive columns deliberately at position 64 and beyond, behind featureless padding, so any recall loss there is the cap's and not a ranking artefact. Invoices carries two columns that are both correct for the same rule. Shipments carries a reversed multi-slot binding, which is structurally valid and semantically wrong. Sensors and employees carry family and decoy cases. Metrics now attribute every miss to the stage that lost it, retrieval or judge, because a recall figure on its own says quality fell without saying where to go and look. Two things this immediately surfaced, and I have left both as they are rather than tuning the fixture around them: The universal not-null rule is not retrieved on four of the eight tables. It carries no column-specific wording, so against 82 rules it loses to eighty more specific ones. Six of the seven current recall misses are this. The judge prompt spends a paragraph instructing the model to apply not-null broadly, which it cannot do for a rule that was never a candidate. The 64-column cap costs one answer of eight on the wide table, but four of the five excluded columns' rules were retrieved anyway via other columns, so that number is not yet a trustworthy estimate of the cap's real cost. It needs the live tier. Baseline: precision 0.891, recall 0.854, retrieval recall 0.889 over 48 labelled answers, five planted wrong-column bindings detected. One sensitivity test was quietly vacuous after the corpus grew: it dropped the first oracle entry, which is now a rule retrieval no longer surfaces, so the injection changed nothing and the test passed anyway. It now names the rule it removes and asserts the baseline really did return it.
…st-eval Tier 2 runs the same fixtures and the same scorer against real endpoints, so unlike Tier 1 it responds to a change in the judge's prompt. Both tiers now come through one assembly function and differ only in the WorkspaceClient they are handed, so they cannot drift apart and stop being comparable. It lives in its own directory rather than behind a marker: app/pyproject.toml sets --strict-markers with no markers declared, and more to the point make app-test should never be one forgotten -m away from spending tokens. The target ignores the directory outright and make app-test-eval is the only way in, gated again by DQX_EVAL_LIVE=1 so a checkout on a machine that happens to hold credentials still costs nothing. Quality is reported rather than asserted until a baseline is recorded from a real run, since an absolute precision threshold on model output is brittle and gets muted within a month. What is asserted does not depend on model quality: every table reaches available=True, no suggestion names a column off the table, every multi-slot mapping is complete with distinct columns, and the 90-column fixture embeds exactly MAX_RETRIEVAL_COLUMNS query texts. The first three are the post-process gates holding against real model output rather than a canned oracle, which is the part Tier 1 cannot check. The cap experiment runs the wide table twice, raising MAX_RETRIEVAL_COLUMNS the second time and printing the delta. It is a module-level constant with no injectable seam, which is the one case AGENTS.md permits patching. Also fixes two ways the metrics lied on the live path. Retrieval coverage can only be measured by reading the outgoing judge prompt, which only the replay client can do, so live runs reported retrieval_recall as 0.000 — indistinguishable from having retrieved nothing. It now reports n/a. For the same reason a live miss cannot honestly be blamed on retrieval or on the judge, so those go to a separate unattributed bucket instead of being counted as retrieval failures. Verified offline by driving the live module's own code paths with the replay client: it reproduces the Tier 1 numbers exactly, so everything except the real endpoint call is exercised. The live run itself is next.
… run exposed
Ran the live tier on field-eng against both the shipped default judge and Claude
Opus 4.7, embeddings on databricks-gte-large-en in both cases.
precision recall f1 p@10 violations golden
databricks-gpt-5-4-nano 0.456 0.854 0.594 0.500 3 45s
databricks-claude-opus-4-7 0.469 0.938 0.625 0.500 0 87s
Claude finds more of the right answers and fell for none of the seven planted
wrong-column bindings that post-process cannot catch, where nano fell for three.
It costs roughly twice the wall clock.
The retrieval column cap now has a price. On the 90-column fixture, reproduced
across two Claude runs, MAX_RETRIEVAL_COLUMNS = 64 costs a quarter of the recall
(0.750 against 1.000 raised to 90) and does not buy precision back. The two
answers it loses are the labelled columns past position 64.
With nano the cap could not be priced at all, and that is a live defect worth
more than the cap question. The judge's output size scales with column count
while max_tokens is a fixed 2048, so past roughly fifty columns it truncates
mid-JSON and the user gets no suggestions whatsoever:
columns 10 20 30 45 64 90
tokens 1174 1599 1344 1831 2048 (cap) 2048 (cap)
usable yes yes yes yes NO NO
It is also mis-reported. AIGateway raises its budget-specific error only when the
content is empty and finish_reason is length; this endpoint returns non-empty
truncated JSON with finish_reason unset, so the user is told the response was
unparsable rather than that the answer did not fit. MAX_RETRIEVAL_COLUMNS does
not mitigate it either — that bounds the retrieval queries, while the judge
prompt still receives every column.
Two defects in my own tests, both found by running for real:
The cap experiment scored a degraded result as recall 0.000 and printed a
meaningless +0.000 delta, because it never checked availability. It now asserts
it and names the measured cause. That is precisely the false green this harness
is supposed to prevent, so it is worth being explicit about.
The baseline comparison was endpoint-blind: the first Claude run cleared a
baseline recorded on nano and reported a pass. Baselines are now keyed by
endpoint pair and the check skips when there is no baseline for the pair in use.
Also correcting a claim I made earlier. I wrote that distractor corpus rules can
never become a prediction; that holds only for the replay tier, where the oracle
bounds the output. The live judge suggests them readily, including a shipment
weight rule bound to parcel_count.
One caveat on the precision figures, and it is this fixture's fault rather than
the model's. The answer key labels the rules a table must have, while the judge
is instructed to apply universal checks to every column a reasonable analyst
would consider required, so defensible not-null suggestions on dispatched_at,
joined_on and similar are counted as false positives. Recall is trustworthy
as-is; precision understates real quality until either every defensible mapping
is labelled or a human judges the actual output.
… precision Two changes from reviewing this before pushing. The live suite is now tests/ai_eval, run by make app-ai-eval. Calling it an integration suite was wrong: it touches no Unity Catalog, no Spark, no Lakebase and no deployed app, every data source is still a double, and what it needs is serving endpoints rather than a workspace. What it produces is a measurement against a statistical baseline, not a wiring check, which is a different kind of test and deserves its own name. It also leaves tests/integration free for a real Studio integration suite later. The baseline now gates on recall only and reports precision without asserting on it. Precision is the noisier metric here, swinging 0.469 to 0.489 across runs of the same fixture, and it is also the less meaningful one, because the answer key labels the rules a table must have while the judge is told to apply universal checks broadly, so defensible suggestions count against it. Recall reproduced exactly at 0.938 across two runs. Gating on precision would have bought a flaky nightly and no signal. Verified after the rename: make fmt clean, make app-test 3856 passed with ai_eval excluded, explicit path still collects it, and it skips without DQX_EVAL_LIVE=1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch brings DQX Studio from the thinner Studio shell on
mainto the full dqlake-parity product (~84 non-merge commits, ~757 files). Relative tomain, it adds the governed rule lifecycle, monitored-table / collection workflows, equal-rule-weight scoring, Results/Genie, RBAC, Marketplace, demo seed, and a complete Studio docs section.Platform & governance
Quality results
Authoring & AI
Docs, demo & polish
Resolves
#1296
Test plan
docs/dqx) — especially quality-score, cross-table, permissionsmainand resolve conflicts (branch is ~88 ahead / ~30 behind)