Skip to content

Add is_valid_url row-level check #1457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ghanse wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add is_valid_url row-level check #1457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0874fe5
Add is_valid_url row-level check
ghanse Aug 12, 2026
a69eeaa
Merge branch 'main' into ghanse/is-valid-url
ghanse Aug 12, 2026
917cbec
Merge branch 'main' into ghanse/is-valid-url
ghanse Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/dqx/docs/reference/quality_checks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ You can also define your own custom checks in Python (see [Creating custom check
| `is_older_than_col2_for_n_days` | Checks whether the values in one input column are at least N days older than the values in another column. | `column1`: first column to check (can be a string column name or a column expression); `column2`: second column to check (can be a string column name or a column expression); `days`: number of days; `negate`: if the condition should be negated |
| `regex_match` | Checks whether the values in the input column match a given regex. | `column`: column to check (can be a string column name or a column expression); regex: regex to check; `negate`: if the condition should be negated (true) or not |
| `is_valid_email` | Checks whether the values in the input column have valid email address format. | `column`: column to check (can be a string column name or a column expression) |
| `is_valid_url` | Checks whether the values in the input column are valid URLs per the RFC 3986 absolute-URI grammar. A scheme is required, so relative references such as `/path` or `example.com` are rejected. Any syntactically valid scheme is accepted, so `s3://`, `ftp://`, `mailto:` and `urn:` are valid alongside `http://` and `https://`. This validates URL syntax only, not safety: script-bearing schemes such as `javascript:alert(1)` and inline `data:` payloads are syntactically valid and pass, so do not use this check to sanitize untrusted input.| `column`: column to check (can be a string column name or a column expression) |
| `has_valid_string_case` | Checks whether string values match the requested letter case: `upper` requires all alphabetic characters to be uppercase; `lower` requires all alphabetic characters to be lowercase; `title` requires the first character of each space-delimited word to be uppercase; `sentence` requires each period-delimited segment's first non-whitespace character to be uppercase. | `column`: column to check (can be a string column name or a column expression); `case`: one of `upper`, `lower`, `title`, or `sentence` |
| `is_valid_national_id` | Checks whether the values in the input column are valid national identification numbers (e.g., US Social Security Numbers) for the given country. | `column`: column to check (can be a string column name or a column expression); `country`: ISO 3166 alpha-2 country code (optional, default: `US`) |
| `is_valid_uuid` | Checks whether the values in the input column are valid UUIDs (RFC 9562, canonical 8-4-4-4-12 hyphenated hex form). By default validates the shape only; set `strict` to also enforce the version nibble (1-8) and variant bits per RFC 9562. | `column`: column to check (can be a string column name or a column expression); `strict`: if True, also validate the version nibble (1-8) and variant bits (8/9/a/b) per RFC 9562 (default: False) |
Expand Down Expand Up @@ -545,6 +546,13 @@ For brevity, the `name` field in the examples is omitted and it will be auto-gen
arguments:
column: col1

# is_valid_url check
- criticality: error
check:
function: is_valid_url
arguments:
column: col1

# is_valid_uuid check
- criticality: error
check:
Expand Down Expand Up @@ -1345,6 +1353,13 @@ checks = [
column="col1"
),

# is_valid_url check
DQRowRule(
criticality="error",
check_func=check_funcs.is_valid_url,
column="col1"
),

# is_valid_uuid check
DQRowRule(
criticality="error",
Expand Down
61 changes: 61 additions & 0 deletions src/databricks/labs/dqx/check_funcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,21 @@
_EMAIL_QTEXT = r"[\x21\x23-\x5B\x5D-\x7E]" # printable ASCII except '"' (0x22) and '\' (0x5C)
_EMAIL_QPAIR = r"\\[\x09\x20-\x7E]" # quoted-pair: '\' + VCHAR or WSP; valid only inside a quoted part

# URL helpers (RFC 3986 absolute-URI grammar). Every repetition below is over alternatives whose first
# characters are disjoint ('%' starts only pct-encoded, '/' only a new path segment), so matching is
# deterministic and there is no catastrophic backtracking; ReDoS-safe.
_URL_SCHEME = r"[A-Za-z][A-Za-z0-9+.\-]*" # RFC 3986 §3.1
_URL_PCT_ENCODED = r"%[0-9A-Fa-f]{2}" # RFC 3986 §2.1
_URL_UNRESERVED_SUB_DELIMS = r"[A-Za-z0-9\-._~!$&'()*+,;=]" # unreserved (§2.3) + sub-delims (§2.2)
_URL_USERINFO = rf"(?:{_URL_UNRESERVED_SUB_DELIMS}|{_URL_PCT_ENCODED}|:)*" # §3.2.1
# reg-name (§3.2.2) also covers IPv4address, since digits and '.' are unreserved.
_URL_HOST = rf"(?:\[[A-Fa-f0-9:.]+\]|(?:{_URL_UNRESERVED_SUB_DELIMS}|{_URL_PCT_ENCODED})*)"
_URL_AUTHORITY = rf"(?:{_URL_USERINFO}@)?{_URL_HOST}(?::\d*)?" # §3.2
_URL_PCHAR = rf"(?:{_URL_UNRESERVED_SUB_DELIMS}|{_URL_PCT_ENCODED}|[:@])" # §3.3
_URL_PATH_ABEMPTY = rf"(?:/{_URL_PCHAR}*)*" # §3.3; each iteration consumes at least the '/'
_URL_PATH_NO_AUTHORITY = rf"(?:/?{_URL_PCHAR}+{_URL_PATH_ABEMPTY})?" # path-absolute / rootless / empty
_URL_QUERY_OR_FRAGMENT = rf"(?:{_URL_PCHAR}|[/?])*" # §3.4, §3.5

# Curated aggregate functions for data quality checks
# These are univariate (single-column) aggregate functions suitable for DQ monitoring
# Maps function names to human-readable display names for error messages
Expand Down Expand Up @@ -118,6 +133,20 @@ class DQPattern(Enum):
# 000/666/9xx (9xx covers ITINs), group 00, serial 0000. Anchored, fixed-width; ReDoS-safe.
SSN_US = r"\A(?!000|666|9\d{2})\d{3}([- ]?)(?!00)\d{2}\1(?!0000)\d{4}\z"

# RFC 3986 §4.3 absolute-URI: scheme ":" hier-part [ "?" query ] [ "#" fragment ]. A scheme is
# required, so relative references ("/path", "example.com") are rejected. Any syntactically valid
# scheme is accepted, which includes non-network schemes such as "javascript:" and "data:" - this
# validates URL *syntax*, not safety. Note that RFC 3986 permits an empty host ("file:///path"),
# so host presence is not enforced here.
# \A...\z anchors (not ^...$) so a trailing newline is rejected under Java regex - see IPV4_ADDRESS.
URL = (
rf"\A{_URL_SCHEME}:"
rf"(?://{_URL_AUTHORITY}{_URL_PATH_ABEMPTY}|{_URL_PATH_NO_AUTHORITY})"
rf"(?:\?{_URL_QUERY_OR_FRAGMENT})?"
rf"(?:#{_URL_QUERY_OR_FRAGMENT})?"
rf"\z"
)

# Canonical UUID form per RFC 9562: 8-4-4-4-12 hex groups. UUID validates the shape
# only, so RFC-defined Nil/Max sentinels and legacy variant GUIDs pass; UUID_STRICT
# also pins the version nibble to 1-8 and variant bits to 8/9/a/b. Anchored, fixed-width; ReDoS-safe.
Expand Down Expand Up @@ -1165,6 +1194,38 @@ def is_valid_email(column: str | Column) -> Column:
return _matches_pattern(column, DQPattern.EMAIL_ADDRESS)


@register_rule("row")
def is_valid_url(column: str | Column) -> Column:
"""Checks whether the values in the input column are valid URLs.

Validates against the RFC 3986 §4.3 *absolute-URI* grammar: *scheme ":" hier-part* with an
optional *"?" query* and *"#" fragment*. A scheme is required, so relative references such as
*/path* or *example.com* are rejected, and reserved characters must be percent-encoded to be
accepted inside a path, query, or fragment.

Any syntactically valid scheme is accepted, which keeps non-network URLs such as *s3://*,
*ftp://*, *mailto:* and *urn:* valid alongside *http://* and *https://*. Two consequences are
worth noting:

* This validates URL *syntax*, not safety or reachability. Script-bearing schemes
(*javascript:alert(1)*) and inline payloads (*data:text/plain,hello*) are syntactically valid
URLs and pass. Do not rely on this check to sanitize untrusted input before rendering or
fetching it; gate the scheme explicitly for that, for example with *is_in_list* on an extracted
scheme column or a *sql_expression* check.
* RFC 3986 permits an empty host, so *file:///path* passes. Host presence is not enforced.

Validation is purely syntactic: it does not verify that the host resolves or that the resource
exists. Null values will pass the check with no violation reported.

Args:
column: column to check; can be a string column name or a column expression

Returns:
Column object for condition
"""
return _matches_pattern(column, DQPattern.URL)


@register_rule("row")
def is_valid_national_id(column: str | Column, country: str = "US") -> Column:
"""Checks whether the values in the input column are valid national identification
Expand Down
31 changes: 28 additions & 3 deletions tests/integration/test_apply_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6002,7 +6002,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"col7: map<string, int>, col8: struct<field1: int>, col10: int, col11: string, "
"col_ipv4: string, col_ipv6: string, col_json_str: string, col_json_str2: string, "
"col_email: string, col_uuid: string, col_ssn: string, col_country: string, col_currency: string, "
"col_subdivision: string, col_language: string"
"col_subdivision: string, col_language: string, col_url: string"
)
test_df = spark.createDataFrame(
[
Expand All @@ -6028,6 +6028,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"USD",
"US-CA",
"en",
"https://example.com/a",
],
[
"val2",
Expand All @@ -6051,6 +6052,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
],
[
"val3",
Expand All @@ -6074,6 +6076,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
],
],
schema,
Expand Down Expand Up @@ -6121,6 +6124,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"USD",
"US-CA",
"en",
"https://example.com/a",
None,
None,
],
Expand All @@ -6146,6 +6150,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
None,
None,
],
Expand All @@ -6171,6 +6176,7 @@ def test_apply_checks_all_row_checks_as_yaml_with_streaming(ws, make_schema, mak
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
None,
None,
],
Expand Down Expand Up @@ -6326,7 +6332,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"col7: map<string, int>, col8: struct<field1: int>, col10: int, col11: string, "
"col_ipv4: string, col_ipv6: string, col_json_str: string, col_json_str2: string, "
"col_email: string, col_uuid: string, col_ssn: string, col_country: string, col_currency: string, "
"col_subdivision: string, col_language: string"
"col_subdivision: string, col_language: string, col_url: string"
)
test_df = spark.createDataFrame(
[
Expand All @@ -6352,6 +6358,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"USD",
"US-CA",
"en",
"https://example.com/a",
],
[
"val2",
Expand All @@ -6375,6 +6382,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
],
[
"val3",
Expand All @@ -6398,6 +6406,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
],
],
schema,
Expand Down Expand Up @@ -6433,6 +6442,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"USD",
"US-CA",
"en",
"https://example.com/a",
None,
None,
],
Expand All @@ -6458,6 +6468,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
None,
None,
],
Expand All @@ -6483,6 +6494,7 @@ def test_apply_checks_all_checks_as_yaml(ws, spark):
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
None,
None,
],
Expand Down Expand Up @@ -7274,6 +7286,12 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
column="col_json_str2",
check_func_kwargs={"schema": "STRUCT<a: STRING, b: STRING>"},
),
# is_valid_url check
DQRowRule(
criticality="error",
check_func=check_funcs.is_valid_url,
column="col_url",
),
# is_valid_national_id check
DQRowRule(
criticality="error",
Expand Down Expand Up @@ -7315,7 +7333,8 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"col1: string, col2: int, col3: int, col4 array<int>, col5: date, col6: timestamp, "
"col7: map<string, int>, col8: struct<field1: int>, col10: int, col11: string, "
"col_ipv4: string, col_ipv6: string, col_json_str: string, col_json_str2: string, col_ssn: string, "
"col_country: string, col_currency: string, col_subdivision: string, col_language: string"
"col_country: string, col_currency: string, col_subdivision: string, col_language: string, "
"col_url: string"
)
test_df = spark.createDataFrame(
[
Expand All @@ -7339,6 +7358,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"USD",
"US-CA",
"en",
"https://example.com/a",
],
[
"val2",
Expand All @@ -7360,6 +7380,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
],
[
"val3",
Expand All @@ -7381,6 +7402,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
],
],
schema,
Expand Down Expand Up @@ -7414,6 +7436,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"USD",
"US-CA",
"en",
"https://example.com/a",
None,
None,
],
Expand All @@ -7437,6 +7460,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"EUR",
"GB-ENG",
"en",
"https://sub.example.org/p?q=1",
None,
None,
],
Expand All @@ -7460,6 +7484,7 @@ def test_apply_checks_all_checks_using_classes(ws, spark):
"GBP",
"DE-BY",
"de",
"ftp://files.example.org/f.txt",
None,
None,
],
Expand Down
Loading
Loading