Skip to content

Feature/categorical values distribution #1450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
IvannKurchenko wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Feature/categorical values distribution #1450

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0dd515c
function doc draft
IvannKurchenko Aug 8, 2026
60bbf7d
Documentation refined
IvannKurchenko Aug 9, 2026
9940438
documentation polishing
IvannKurchenko Aug 9, 2026
1b81416
validation implementation and tests
IvannKurchenko Aug 9, 2026
c7d8502
draft implementation
IvannKurchenko Aug 9, 2026
cec8b5b
tests fixes
IvannKurchenko Aug 11, 2026
f499bb2
linting and formatting
IvannKurchenko Aug 11, 2026
23b3460
Merge branch 'main' into feature/categorical_values_distribution
mwojtyczka Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions docs/dqx/docs/reference/quality_checks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1942,6 +1942,7 @@ You can also define your own custom dataset-level checks (see [Creating custom c
| `has_no_gaps_per_time_window` | Dataset check that flags gaps in a time series, i.e. time windows of a given size that contain no rows between windows that do. The violation is reported on the boundary row before each interior gap. | `column`: timestamp or date column (can be a string column name or a column expression); `window_minutes`: size of the time window in minutes that defines the expected data grain (for example 1440 for daily); `group_by`: optional list of columns or column expressions to detect gaps independently within each group; `trailing_gap`: (optional) if `true`, also flags the last present window (per group) when it ends more than one window before `curr_timestamp`, so missing recent data is caught at the tail of the series (defaults to `false`); `curr_timestamp`: (optional) current timestamp column used to anchor trailing-gap detection, only used when `trailing_gap` is `true` (if not provided, current_timestamp() function is used) |
| `has_valid_schema` | Schema check that validates whether the DataFrame schema matches an expected schema. In non-strict mode, validates that all expected columns exist with compatible types (allows extra columns). In strict mode, validates exact schema match (same columns, same order, same types) for all columns by default or for all columns specified in `columns`. This check is applied at the dataset level and reports schema violations for all rows in the DataFrame when incompatibilities are detected. All columns in the `exclude_columns` list will be ignored even if the column is present in the `columns` list. | `expected_schema`: (optional) expected schema as a DDL string (e.g., "id INT, name STRING") or StructType object; `ref_df_name`: (optional) name of the reference DataFrame to load the schema from (dictionary of DataFrames can be passed when applying checks); `ref_table`: (optional) fully qualified reference table name to load the schema from (e.g. "catalog.schema.table"); exactly one of `expected_schema`, `ref_df_name`, or `ref_table` must be provided; `columns`: (optional) list of columns to validate (if not provided, all columns are considered); `strict`: (optional) whether to perform strict schema validation (default: False) - False: validates that all expected columns exist with compatible types, True: validates exact schema match; `exclude_columns`: (optional) list of columns to ignore during validation (if not provided, all columns are considered); |
| `has_no_outliers` | Checks whether the values in the input column contain any outliers. This function implements a median absolute deviation (MAD) algorithm to find outliers. | `column`: column of type numeric to check (can be a string column name or a column expression); |
| `is_in_distribution` | Checks whether the discrete value distribution of the column is within a given distance of an expected distribution, using the **Total Variation Distance (TVD)** of probability measures. Supplements `is_in_list` at the dataset level by validating not only that values stay within an enumerated set, but also that they follow the expected proportions. Supported column types: Boolean, Char, String, Byte, Short, Integer, Long, and Date. NULL values in the checked column are skipped. When the sum of the expected distribution values is strictly less than 1, the missing mass is placed in an internal `residual` bucket, and any column values not listed as explicit keys are aggregated into the same bucket. | `column`: column to check (can be a string column name or a column expression); `distribution`: expected distribution as a dict mapping literal values to their expected proportions; `distance`: maximum allowed TVD between the actual and expected distributions; `case_sensitive`: (optional) whether string comparisons are case-sensitive (default: `True`); `impute`: (optional) if `True` (default), keys present in the expected distribution but missing from the actual distribution are imputed with `0`; if `False`, the check fails and reports the missing keys |
| `has_no_row_anomalies` | Flags rows that are anomalous according to a trained ML model. The model learns "normal" patterns from your training data; at check time each row is scored (severity percentile 0–100) and optionally enriched with SHAP contributions. Requires a model trained with the anomaly engine first. See [Row Anomaly Detection](#row-anomaly-detection) below for training, full parameters, and usage. | `model_name`: fully qualified model name (e.g. catalog.schema.model_name); `registry_table`: fully qualified registry table (e.g. catalog.schema.model_registry); `threshold`: (optional) severity percentile threshold (default 95); `drift_threshold`: (optional) warn when score distribution drifts from training (None = off); `enable_contributions`: (optional) add SHAP per-feature contributions to `_dq_info` (default True; set False to skip the SHAP cost); `enable_confidence_std`: (optional) add ensemble score std to `_dq_info` (default False); `enable_ai_explanation`: (optional) add an LLM-generated explanation to `_dq_info` (default True; degrades to null if contributions are off or no serving endpoint is reachable); `ai_explanation_llm_model_config`: (optional) Databricks Model Serving endpoint config for the explanation; `redact_columns`: (optional) feature/segment names to keep out of the LLM prompt; `max_groups`: (optional) cap on LLM calls per run (default 500). See [Row Anomaly Detection](/docs/reference/quality_checks#row-anomaly-detection) section for full parameter details. |
| `are_polygons_mutually_disjoint` | Checks whether the polygons in a geometry column are mutually disjoint. Polygons sharing an edge or boundary are considered intersecting. Nulls and invalid geometries are excluded from the check. Requires Databricks runtime 17.1 or above. | `column`: column to check (can be a string column name or a column expression), must contain polygon or multipolygon geometries |
| `is_geo_contains` | Checks if the reference geometry contains each column geometry using `st_contains` with meter-level precision. A geometry A *contains* B when B lies entirely within the interior of A with no boundary points of B on the boundary of A. Points on the shared boundary are not considered contained — use `is_geo_covers` for boundary-inclusive checks. When a convert flag is set to `True`, `try_to_geometry` is applied to parse the input from any supported format (WKT, WKB, EWKT, EWKB). Null values are skipped. Requires Databricks runtime 17.1 or above. | `column`: column to check (can be a string column name or a column expression); `reference_geometry`: reference geometry as a literal WKT/WKB/EWKT/EWKB string or bytes value, or a `Column` expression (e.g. `F.col('col_name')`) — a plain string is always treated as a literal, not a column name; `convert_column`: when `True`, applies `try_to_geometry` to convert the column values to GEOMETRY (default `False`); `convert_reference_geometry`: when `True`, applies `try_to_geometry` to convert the reference geometry to GEOMETRY (default `False`) |
Expand Down Expand Up @@ -2466,6 +2467,50 @@ Complex data types are supported as well.
arguments:
column: col1

# is_in_distribution check — validates the actual value distribution of a categorical
# column against an expected distribution using Total Variation Distance (TVD).
# Fails when TVD(actual, expected) > distance.
- criticality: error
check:
function: is_in_distribution
arguments:
column: status
distribution:
active: 0.7
inactive: 0.2
pending: 0.1
distance: 0.05 # max allowed TVD

# is_in_distribution check — expected distribution intentionally covers only a subset
# of the possible values (sum < 1); case-insensitive comparison. The missing 0.1 mass
# is placed in an internal `residual` bucket; any column values other than A/B/C
# are aggregated into `residual` and compared against that 0.1 expected mass.
- criticality: warn
check:
function: is_in_distribution
arguments:
column: category
distribution:
A: 0.6
B: 0.2
C: 0.1 # remaining 0.1 goes to an internal `residual` bucket for all other values
distance: 0.1
case_sensitive: false

# is_in_distribution check — fail (and enumerate missing keys) if any expected key
# is absent from the actual distribution, instead of imputing it with 0.
- criticality: error
check:
function: is_in_distribution
arguments:
column: tier
distribution:
gold: 0.2
silver: 0.3
bronze: 0.5
distance: 0.05
impute: false # fail and report missing keys instead of imputing with 0

# are_polygons_mutually_disjoint check (geo, requires runtime 17.1+)
- criticality: error
check:
Expand Down Expand Up @@ -3073,6 +3118,47 @@ checks = [
column="col1" # or as expr: F.col("col1")
),

# is_in_distribution check — validates the actual value distribution of a categorical
# column against an expected distribution using Total Variation Distance (TVD).
# Fails when TVD(actual, expected) > distance.
DQDatasetRule(
criticality="error",
check_func=check_funcs.is_in_distribution,
column="status", # or as expr: F.col("status")
check_func_kwargs={
"distribution": {"active": 0.7, "inactive": 0.2, "pending": 0.1},
"distance": 0.05, # max allowed TVD
}
),

# is_in_distribution check — expected distribution intentionally covers only a subset
# of the possible values (sum < 1); case-insensitive comparison. The missing 0.1 mass
# is placed in an internal `residual` bucket; any column values other than A/B/C
# are aggregated into `residual` and compared against that 0.1 expected mass.
DQDatasetRule(
criticality="warn",
check_func=check_funcs.is_in_distribution,
column="category", # or as expr: F.col("category")
check_func_kwargs={
"distribution": {"A": 0.6, "B": 0.2, "C": 0.1}, # remaining 0.1 goes to internal `residual` bucket
"distance": 0.1,
"case_sensitive": False,
}
),

# is_in_distribution check — fail (and enumerate missing keys) if any expected key
# is absent from the actual distribution, instead of imputing it with 0.
DQDatasetRule(
criticality="error",
check_func=check_funcs.is_in_distribution,
column="tier", # or as expr: F.col("tier")
check_func_kwargs={
"distribution": {"gold": 0.2, "silver": 0.3, "bronze": 0.5},
"distance": 0.05,
"impute": False, # fail and report missing keys instead of imputing with 0
}
),

# are_polygons_mutually_disjoint check (geo, requires runtime 17.1+)
DQDatasetRule(
criticality="error",
Expand Down
Loading