-
Notifications
You must be signed in to change notification settings - Fork 35
test: [expected failures] base fixtures needing implementation fixes #163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: mainline
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Parameter type names are case-sensitive in base (case-insensitivity is EXPR-only); 'string' must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: string | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print()" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — same family, allowedValues axis; an unrepresentable allowedValue is dead weight, low blast radius. Matches sweep; twin in PR #161. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # An INT parameter allowedValues entry of 2^63 exceeds the signed 64-bit integer range and must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| allowedValues: | ||
| - 9223372036854775808 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print('{{Param.MyParam}}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS (majority; 1 NOT-GOOD dissent) — the defect is real (both implementations accept 2^63; matches sweep) and rejection is a safe data-integrity adoption (values that can't round-trip int64 storage). Dissent (spec-literalist): the base 2023-09 spec never bounds — int64 exists only in the EXPR doc — so a bignum implementation is arguably conformant and this is a SPEC QUESTION, not an implementation bug. Consensus action: keep parked, but reword the README classification to 'pending base-spec integer-bounds erratum' and make that erratum the promotion gate. Also: README cites accept twin jobs/2.3--int-value-above-int64-max.invalid.test.yaml as 'in the main suite' — it's on PR #161's branch (and per that PR's review, it should move HERE). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # An INT parameter default of 2^63 exceeds the signed 64-bit integer range and must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| default: 9223372036854775808 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print('{{Param.MyParam}}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — same family, negative bound (-2^63-1). Matches sweep; accept twin 2.3--int-default-int64-min.yaml in PR #161. Same spec-erratum caveat as the family. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # An INT parameter default of -2^63-1 is below the signed 64-bit integer range and must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| default: -9223372036854775809 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print('{{Param.MyParam}}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — same family, maxValue axis. Service note: likeliest of the five to hit real templates (maxValue=2^63 as an infinity sentinel) — flag in any service change notice. Matches sweep; twin in PR #161. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # An INT parameter maxValue of 2^63 exceeds the signed 64-bit integer range and must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| maxValue: 9223372036854775808 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print('{{Param.MyParam}}')" |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — same family, minValue axis; per-field decomposition justified (separate parse paths). Matches sweep; twin in PR #161. Inventory note: negative-side rejects (minValue/maxValue/allowedValues below -2^63) are uncovered — likely same code path, but note it. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # An INT parameter minValue of 2^63 exceeds the signed 64-bit integer range and must be rejected. | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| minValue: 9223372036854775808 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - "-c" | ||
| - "print('{{Param.MyParam}}')" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # Proposed conformance fixtures (expected failures) — base/job_templates | ||
|
|
||
| Every fixture in this directory is parked because it **fails against at least | ||
| one current reference implementation**. Placement is kind-level | ||
| (`<component>/<kind>/proposed/`) so promotion is a mechanical move up one | ||
| directory once the underlying issue is resolved. The conformance runner does | ||
| not scan `proposed/` directories. | ||
|
|
||
| Verification environment: `openjd` CLI = openjd-rs release build from | ||
| upstream/main (post range-cap fix); cross-checks with openjd-model-for-python | ||
| 0.11.x via `decode_job_template`. | ||
|
|
||
| ## INT 64-bit boundary rejects | ||
|
|
||
| Spec §2.3 types these fields as `<integer>` with **no stated bound** — the | ||
| base 2023-09 document never mentions 64-bit anywhere; the int64 range exists | ||
| only in the Expression Language type table (an EXPR document). Both reference | ||
| implementations nonetheless *store* INT as a signed 64-bit integer, so | ||
| accepted values outside [-2^63, 2^63-1] cannot round-trip and silently | ||
| corrupt. | ||
|
|
||
| | Fixture | Construct | Observed | | ||
| |---|---|---| | ||
| | `2.3--int-default-above-int64-max.invalid.yaml` | `default: 9223372036854775808` (2^63) | accepted by **both** (openjd-rs and python) | | ||
| | `2.3--int-minvalue-above-int64-max.invalid.yaml` | `minValue: 9223372036854775808` | accepted by both | | ||
| | `2.3--int-maxvalue-above-int64-max.invalid.yaml` | `maxValue: 9223372036854775808` | accepted by both | | ||
| | `2.3--int-allowedvalues-above-int64-max.invalid.yaml` | `allowedValues: [9223372036854775808]` | accepted by both | | ||
| | `2.3--int-default-below-int64-min.invalid.yaml` | `default: -9223372036854775809` (-2^63-1) | accepted by both | | ||
|
|
||
| Classification: **spec question first, then implementation fix** — because the | ||
| base spec states no integer bound, a bignum implementation accepting these is | ||
| arguably conformant as the text stands. Promotion is gated on a base-spec | ||
| integer-bounds erratum (pin `<integer>` to int64 in §2.3); once that lands, | ||
| these become straightforward validation-bug pins, one branch per field. The | ||
| matching accept-twins at 2^63-1 and -2^63 are added by the base-gaps PR | ||
| (`job_templates/2.3--int-*-int64-max.yaml` etc.). | ||
|
|
||
| Related, different axis: the *supplied value* case lives in | ||
| `../../jobs/proposed/2.3--int-value-above-int64-max.invalid.test.yaml` — that | ||
| one is **Python-only** (openjd-rs rejects a supplied 2^63 at job creation; | ||
| openjd-model 0.11.x `create_job` accepts it). | ||
|
|
||
| ## Type-name case sensitivity | ||
|
|
||
| | Fixture | Construct | Observed | | ||
| |---|---|---| | ||
| | `2--type-lowercase.invalid.yaml` | `type: string` in base (no `extensions:`) | openjd-rs: **accepted**; python: rejected ("Input tag 'string' ... does not match any of the expected tags") | | ||
|
|
||
| Spec §2: `type: "STRING"` is a literal, and the spec states type names become | ||
| case-insensitive only "When the `EXPR` extension is enabled" (see RFC 0007). | ||
| Base is case-sensitive. | ||
|
|
||
| Classification: **openjd-rs implementation bug** (unconditional | ||
| case-insensitive type parsing); python is spec-conformant. Note: the | ||
| param-types expected-failures PR previously carried an identical pin | ||
| (`2--type-lowercase-string.invalid.yaml`); it was dropped in favor of this | ||
| one — this fixture is the single copy. |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS (5/5 with the same caveat) — spec: §1.2.1 ("each constraint must become more constrained", Job Template processed last). The literal reading supports rejection and the README honestly offers the alternative (intersection-merge) reading — this is closer to a spec decision than a clean mandate; the fixture may be invalidated by a spec revision rather than an implementation fix. Service note: adoption flips CreateJob accept→reject for queue-environment parameter merging (BealineDataPlane grafts queue-env param definitions onto customer templates), and the un-scoped constraint-OMISSION case is the real-world hazard — gate promotion on the spec decision plus a service deprecation path. Matches sweep (fails both). Accept twin: mainline 1.2.1--constraint-narrowing-int-range. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Spec 1.2.1: each constraint must become MORE constrained in template processing order. | ||
| # The job template (processed last) widens the environment template's INT range, so the | ||
| # merge must be rejected. The default satisfies both ranges, isolating the widening rule. | ||
| template: | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: Count | ||
| type: INT | ||
| minValue: 0 | ||
| maxValue: 1000 | ||
| default: 50 | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - -c | ||
| - print(r'Count={{Param.Count}}') | ||
| environments: | ||
| - specificationVersion: environment-2023-09 | ||
| parameterDefinitions: | ||
| - name: Count | ||
| type: INT | ||
| minValue: 10 | ||
| maxValue: 100 | ||
| environment: | ||
| name: TestEnv | ||
| variables: | ||
| COUNT: '{{Param.Count}}' |
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quorum verdict: GOOD-WITH-NITS — spec: §1.2.1, STRING minLength axis. Same narrow-vs-intersection ambiguity and same queue-env merge hazard as the int-range twin; good isolation (default satisfies both ranges). Accept twin: mainline 1.2.1--constraint-narrowing-minlength. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Spec 1.2.1: each constraint must become MORE constrained in template processing order. | ||
| # The job template (processed last) relaxes the environment template's minLength, so the | ||
| # merge must be rejected. The default satisfies both, isolating the widening rule. | ||
| template: | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: Value | ||
| type: STRING | ||
| minLength: 2 | ||
| default: test-value | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - -c | ||
| - print(r'Value={{Param.Value}}') | ||
| environments: | ||
| - specificationVersion: environment-2023-09 | ||
| parameterDefinitions: | ||
| - name: Value | ||
| type: STRING | ||
| minLength: 5 | ||
| environment: | ||
| name: TestEnv | ||
| variables: | ||
| VAL: '{{Param.Value}}' |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # A supplied INT parameter value of 2^63 exceeds the signed 64-bit integer range and must be rejected at job creation. | ||
| template: | ||
| specificationVersion: jobtemplate-2023-09 | ||
| name: TestJob | ||
| parameterDefinitions: | ||
| - name: MyParam | ||
| type: INT | ||
| steps: | ||
| - name: Step1 | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: | ||
| - -c | ||
| - print(r'OUTPUT:{{Param.MyParam}}') | ||
| parameters: | ||
| MyParam: 9223372036854775808 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Proposed conformance fixtures (expected failures) — base/jobs | ||
|
|
||
| Every fixture in this directory is parked because it **fails against at least | ||
| one current reference implementation**. Placement is kind-level | ||
| (`<component>/<kind>/proposed/`) so promotion is a mechanical move up one | ||
| directory. The conformance runner does not scan `proposed/` directories. | ||
|
|
||
| ## Merge widening (§1.2.1) | ||
|
|
||
| Spec §1.2.1: "each constraint must become more constrained as a subsequent | ||
| definition is merged with the previous ones in processing order" (Job | ||
| Template last). These fixtures have a Job Template that *widens* the | ||
| Environment Template's constraint while the default satisfies both ranges, | ||
| isolating the direction rule from the merged-consistency rule. | ||
|
|
||
| | Fixture | Construct | Observed | | ||
| |---|---|---| | ||
| | `1.2.1--constraint-widening-int-range.invalid.test.yaml` | env `[10,100]`, job template `[0,1000]` | accepted by both; job runs | | ||
| | `1.2.1--constraint-widening-minlength.invalid.test.yaml` | env `minLength: 5`, job template `minLength: 2` | accepted by both; job runs | | ||
|
|
||
| Cross-check: `openjd.model.merge_job_parameter_definitions` returns the | ||
| *intersection* (`minValue=10, maxValue=100`) rather than rejecting — both | ||
| implementations merge by intersection and never enforce the narrowing | ||
| direction. | ||
|
|
||
| Classification: **spec decision needed, then (possibly) implementation fix**. | ||
| The stated sentence supports rejection on a literal reading, but both | ||
| implementations implement intersection semantics, and enforcing | ||
| reject-on-widening is a behavioral change for any scheduler that merges | ||
| externally-supplied environment parameter definitions into customer templates | ||
| (e.g. queue environments). Until the spec either affirms the narrowing rule | ||
| or is revised to define intersection semantics, these stay parked. | ||
|
|
||
| ## INT 64-bit supplied value (Python-only) | ||
|
|
||
| | Fixture | Construct | Observed | | ||
| |---|---|---| | ||
| | `2.3--int-value-above-int64-max.invalid.test.yaml` | supplied INT parameter value `9223372036854775808` (2^63) | openjd-rs: **rejects at job creation (fixture passes)**; python (openjd-model 0.11.x `create_job`): **accepts** | | ||
|
|
||
| Moved here from the base-gaps PR's live `jobs/` directory: the base spec | ||
| states no int64 bound on `<integer>` (see the job_templates/proposed README), | ||
| and the reference Python implementation accepts the value, so as a live | ||
| fixture it failed the reference implementation. Same promotion gate as the | ||
| template-side int64 family: a base-spec integer-bounds erratum, plus the | ||
| openjd-model fix. Its accept twin (`jobs/2.3--int-value-int64-max.test.yaml`, | ||
| supplied 2^63-1 resolves exactly) is in the base-gaps PR and passes both. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quorum verdict: GOOD (5/5, one dedupe action) — spec: §2 (base type names case-sensitive; case-insensitivity is EXPR-gated per RFC 0007 — the BNF literal plus the §2 sentence's negative implication is solid). Matches sweep (openjd-rs accepts, Python rejects) and the README states both. Action: this duplicates PR #166's base/job_templates/proposed/2--type-lowercase-string.invalid.yaml — same construct, same defect, two paths; keep one (or differentiate on the type token).