test: Add conformance fixtures for every coercible 2023-09 field - #179
Open
leongdl wants to merge 6 commits into
Open
test: Add conformance fixtures for every coercible 2023-09 field#179leongdl wants to merge 6 commits into
leongdl wants to merge 6 commits into
Conversation
…quirements
Three fields that resolve at job creation had no fixture asserting the resolved
value: chunks.defaultTaskCount, chunks.targetRuntimeSeconds, and
hostRequirements.amounts[].min. All three are annotated @fmtstring without
[host], so a format string in the template must be resolved before the value is
used. No fixture in the suite wrote any of them as a format string.
This is not hypothetical. A service reading chunks.defaultTaskCount out of the
pre-resolution template rejected valid task-chunking jobs in production, because
it saw the literal "{{Param.ChunkSize}}" where the resolved job carries an int.
Added to the live suite, passing both reference implementations:
- TASK_CHUNKING/jobs/default-task-count-format-string.test.yaml asserts the
resolved value through the chunk boundaries it produces.
- TASK_CHUNKING/jobs/target-runtime-seconds-format-string.test.yaml resolves to
0, where section 3.4.1.5 fixes the scheduler's behavior, so the boundaries are
deterministic.
- FEATURE_BUNDLE_1/jobs/3.3.1--amount-min-format-string-resolves-to-non-number
.invalid.test.yaml covers a min that resolves to a non-numeric string.
Added under proposed/, believed spec-correct and currently failing the Rust
implementation. Both are the same defect shape: a bound that cannot be checked
at decode because the value is a format string, deferred correctly, then never
re-checked after resolution.
- TASK_CHUNKING/jobs/proposed/ covers a defaultTaskCount that resolves to 0
against the documented minimum of 1.
- base/jobs/proposed/ covers an anyOf element that resolves to a value violating
the section 3.3.2.2 pattern.
Each fixture was mutation-checked: change the resolved value, leave the
expectations, confirm it fails. The targetRuntimeSeconds fixture was also
mutated to a non-zero resolved value, which engages adaptive chunking and
changes the boundaries, so it pins that field specifically rather than only
defaultTaskCount.
Not covered, and not writable today: the positive case asserting a resolved
amounts[].min, .max or anyOf value is correct. No openjd CLI surfaces resolved
host requirements, and the runner asserts only on stdout and task status, so
those values have no observable effect on a single-host run.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The first commit covered the fields implicated in one production defect. This covers the rest of the surface: every 2023-09 field where the raw template and the resolved job can hold different values. Two classes were missing entirely. The <intstring> and <floatstring> unions. JobIntParameterDefinition and JobFloatParameterDefinition declare default, allowedValues, minValue and maxValue as `<integer> | <intstring>` in base 2023-09 with no extension gate, and IntRangeList and FloatRangeList do the same for their elements. The suite had template-level accept coverage and no job-level coverage of any string-form definition field, so nothing asserted what a string form resolves to. Across 2,417 IntRangeList elements and 1,044 FloatRangeList elements in the suite, not one used the string form. Fields whose resolved value a CLI does not print. cancelation .notifyPeriodInSeconds written as a format string was covered only by a fixture asserting a constant, because neither `openjd run` nor `openjd summary` surfaces the field. WrappedAction.Cancelation.NotifyPeriodInSeconds from RFC 0008 closes it: the reflection variable carries the effective value after resolution, so a wrap script reads back what the job actually got. Added to the live suite, passing both reference implementations: - WRAP_ACTIONS/jobs/wrap-cancelation-notify-period-fmtstring-resolved - base/jobs/1.1.1--resolved-job-name-value, asserting the resolved name field at base level through the implementation's own report of the job it ran. Existing base fixtures assert a parallel substitution into a task argument instead. - base/jobs/1.1--path-default-joined-with-template-dir - base/jobs/2.3--int-param-intstring-default-resolves - base/jobs/1.1--int-intstring-bounds-satisfied and its .invalid pair - base/jobs/1.1--int-intstring-allowedvalues-violation.invalid - FEATURE_BUNDLE_1/jobs/3.3.1--amount-max-format-string-resolves-to-non-number .invalid, the max counterpart to the min case in the previous commit Strengthened one existing fixture. 1.1--validation-valid-path-default-within -template-dir asserted the substring /output against a default of ./output, and /output is a substring of ./output, so an implementation that skipped the template-directory join passed it. Verified by running it with the task hardcoded to print the un-joined value: it passed. Two forbidden entries close that, and the same mutant now fails. Added under proposed/, believed spec-correct and currently failing the Rust implementation: - FEATURE_BUNDLE_1 amounts[].max resolving to 0, against <positivefloat>. The only assertion in the suite that distinguishes max semantics from min. - base attributes[].allOf resolving to an invalid <AttributeCapabilityValue>, the allOf counterpart to the anyOf case in the previous commit. - base IntRangeList and FloatRangeList string-form elements. These carry a spec question rather than a clear defect: <intstring> and <floatstring> are defined only as base-10 string representations, with nothing said about normalization, and the two implementations disagree in opposite directions on different fields. EXPR/jobs/expr1.3.4--float-passthrough already pins the verbatim reading for a parameter default, so a ruling is needed before either it or the float fixture here can be called conformant. Every fixture was run against both CLIs and mutation-checked. Nine mutants, nine caught. The notifyPeriodInSeconds and job-name fixtures were additionally mutated by removing their parameter overrides, so a fixture satisfied by a template default rather than a resolved submitted value would fail. Not covered, and not writable: positive assertions on resolved host-requirement values. No CLI surfaces them, and WRAP_ACTIONS reflection is scoped to <Action> fields, so the eight reflection variables cannot reach host requirements. Also not written: capability name as a format string, which needs a spec decision first, since the spec declares <AmountCapabilityName> "A string" while one implementation resolves it and the other rejects it. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Every fixture whose subject is a field annotated @fmtstring wrote its parameter value as a native integer, so it exercised the format-string layer and left the <intstring> layer untouched. Those are two coercions in series, and the seam between them was untested: an <intstring> parameter default must first parse to a number, and only then can the format string on the field resolve to it. Writing the parameter value in the string form covers both layers with the same assertion, so this is strictly stronger at no cost rather than extra coverage. - chunks.defaultTaskCount now resolves from default '03'. The boundaries 1-3/4-6/7-8 hold only if the string parsed to 3 and the format string then resolved to it. - chunks.targetRuntimeSeconds resolves from default '00'. - The two proposed bound-check fixtures use '0', so the bound must be enforced after both coercions rather than after one. - The wrapped action's notifyPeriodInSeconds takes a submitted '047', chaining three coercions: the string parses to 47, the format string on the action resolves to it, and the reflection variable reports the effective value. Leading zeros are deliberate. A bare '3' would be indistinguishable from a passed-through literal, whereas '03' resolving to 3 can only be a parsed integer, and the fixtures already forbid the un-normalized text. Re-verified on both reference CLIs, unchanged results. Four new mutants, four caught: '03' to '04' fails, '03' to 'abc' fails at parse rather than passing quietly, '00' to '03' engages adaptive chunking and shifts the boundaries, and '047' to '048' fails through the reflection variable. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
leongdl
changed the title
Each fixture asserted a spec requirement without giving a reviewer a way to check
it. References were bare line numbers such as "Template Schemas L1263", which
are not clickable, are not the convention the suite already uses, and appear in
no pre-existing fixture. Three fixtures cited nothing at all.
Every fixture and every proposed/README.md now carries a Spec references block
listing each claim it depends on, with the section number and a line-anchored
link:
§3.4.1.5 L1263 chunks.defaultTaskCount is `<integer> | <intstring>` and @fmtstring
https://github.com/.../2023-09-Template-Schemas.md?plain=1#L1263
Three details behind the format:
`?plain=1` is required. GitHub renders markdown, so a bare `#L1263` on a .md file
does not anchor; only the plain view has line anchors.
Both the section and the line are given. The section is the durable reference if
the spec is re-flowed, and the line is where the claim was actually verified.
Prose now uses §X.Y throughout, matching the 32 existing fixtures that already
cite sections that way.
Citations are verified, not asserted. A checker walks every §X.Y L### pair in the
diff, confirms the cited line really sits under the cited heading by parsing the
spec's headings, and requires a keyword supporting the claim within three lines
of the target. 71 citations, 0 problems. It also confirms every `?plain=1#L`
link has a matching section reference in the same file, so a link can never
drift away from the claim it supports.
One reference was repointed rather than reformatted. The wrap-actions fixture
cited L1776, which sits in the generic §5.3 `<CancelationMethod>` block; the
precise citation for a NOTIFY_THEN_TERMINATE notifyPeriodInSeconds is §5.3.2
L1835. It now cites that, plus §4.3.1 L1630 for the reflection variable and RFC
0008 for the wrap hook variable table.
All 12 live fixtures re-run on both reference CLIs after the rewrite, unchanged
results. Comment-only change; no assertion, template or expectation was touched.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Correcting the previous commit, whose message claimed "71 citations, 0 problems". That was not accurate. Two of the 22 distinct citations pointed at structure rather than at the claim, and my own verifier let them through. §7.4 L2014 is the stage table's header row, `| Stage | When | Known Values | ...`. Both claims cited against it, that format strings without @fmtstring[host] resolve at job creation and that PATH parameter defaults are joined with the job template directory, are on L2017, the Job creation row. Seven uses repointed. The worst instance put the PATH sentence in direct quotation marks attributed to a line that does not contain it. §3.3.2.2 L1054 is the heading `##### 3.3.2.2. <AttributeCapabilityValue>`. Citing a heading tells a reader the section, which the §X.Y already says, and not where the requirement is. The load-bearing line for these fixtures is L1059, "After the format string has been resolved:", which is precisely the post-resolution point they assert. Three uses repointed. The cause was the verifier, not carelessness in transcription. It accepted a supporting keyword within ±3 lines of the target, and L2014 to L2017 is exactly 3, so a citation aimed at a table header scored as correct. A check that loose cannot catch the error it exists to catch. The verifier is now strict on three counts: the keyword must appear ON the cited line rather than near it; a heading is never a valid target; and a table header or separator row is never a valid target, which is exactly what L2014 was. Under those rules all 71 citations pass and every link still carries ?plain=1 and has a matching section reference in the same file. Tightening it also caught a fault in the checker rather than the fixtures: it expected the keyword `range:` at §3.4.1.1 L1110 and §3.4.1.2 L1184, but those lines are the `<IntRangeList>` and `<FloatRangeList>` grammar productions, which is exactly what the fixtures cite them for. The expectation was wrong, not the citation, and I corrected the checker rather than the reference. All 12 live fixtures re-run on both reference CLIs. Comment-only change. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
…e field
Two citations named the field declaration where the claim was about the type
definition. §2.3 L306 is `default: <integer> | <intstring>`, which supports "INT
parameter default is `<integer> | <intstring>`" and does not support
"`<intstring>` is a base-10 string representation"; that sentence is L318. Same
shape for §2.4 L363 against L375 for `<floatstring>`. Three sites repointed, with
their links. The other uses of L306, which cite the field declaration, are
correct and unchanged.
Also reworded the §3.3.2.2 L1059 claim. The line is "After the format string has
been resolved:", so citing it for "the pattern the resolved value must satisfy"
overstated it; the pattern is L1060-L1062. It now reads "constraints apply AFTER
the format string resolves", which is what the line says and is the fixture's
actual point.
The verifier is the recurring problem here, so it changed approach rather than
gaining another tweak. Three versions each failed differently: a ±3-line window
let a table-header citation pass; keying the expectation on the line number meant
a line cited for two different claims was only checked against one, which is
exactly how these two survived; and lifting the claim text to match tokens
against the line produced twelve false positives, because it cannot match
`amounts[].max` against `max:`, flags editorial words no spec line contains, and
cannot tell a nested list item from its parent bullet.
Three different failures from three attempts is a signal about the approach, not
the tuning. Automated claim-to-line matching is not reliable on prose. The
verifier now automates only what is mechanically decidable, and stops pretending
about the rest:
automated section containment, target is not a heading, not a table header or
separator, not a blank line, every link has a matching reference in
the same file, every schema link carries ?plain=1
human whether the line states the claim, printed as a two-column table of
all 23 distinct references beside the spec text
That division is not a concession, it is where the results came from: every real
error in this series was found by a person reading the line, and none by any of
the three checkers. Mechanical checks now pass with 0 problems across 21 files
and 23 distinct references, and I read all 23 pairs.
All 12 live fixtures re-run on both reference CLIs. Comment-only change, and all
three touched fixtures are under proposed/, which CI does not discover.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
jericht
reviewed
Aug 27, 2026
jericht
left a comment
jericht
left a comment
There was a problem hiding this comment.
Overall LGTM just a couple missing test cases
Comment on lines
+23
to
+24
| minValue: '010' | ||
| maxValue: '100' |
There was a problem hiding this comment.
Missed coercion (Finding 1): no FLOAT analog of this fixture.
JobFloatParameterDefinition (§2.4, L363–366) declares minValue/maxValue/allowedValues as <float> | <floatstring> — identical to INT here, no extension gate — but this PR adds the string-form numeric-comparison fixtures only for INT. There is no job-level FLOAT equivalent of 1.1--int-intstring-bounds-satisfied / -violation / -allowedvalues-violation.
Existing FLOAT coverage does not fill it: base/jobs/1.1--float-parameter-string-coercion-valid coerces the submitted value and declares no bounds, and base/job_templates/2.4--float-param-minmax-floatstring.yaml is decode-only with bounds "0.0"/"1.0" that do not diverge lexically from a numeric compare.
This is not blocked by the <floatstring> normalization spec question the proposed range fixtures defer: numeric comparison is normalization-independent (50.0 >= 10.0 regardless of stored form), so a FLOAT bounds/allowedValues fixture can land live exactly like this one. Suggest adding floatstring minValue/maxValue with a lexically-divergent bound (e.g. '010.0') plus an allowedValues numeric-membership case. (A FLOAT default-resolves fixture should stay deferred — it does collide with EXPR/jobs/expr1.3.4--float-passthrough.)
Contributor
Author
There was a problem hiding this comment.
Interesting and we should add it if the spec is missing it. This would need a new test case, not changing this one.
| hostRequirements: | ||
| amounts: | ||
| - name: amount.worker.vcpu | ||
| min: "{{Param.CpuMin}}" |
There was a problem hiding this comment.
Finding 2 (secondary): amount min/max plain string-form numeric comparison is untested.
amounts[].min/.max (§3.3.1 L958/960) are <nonnegativefloat> | <nonnegativefloatstring> and @fmtstring. This fixture covers the @fmtstring→non-number path. The plain string-form path is not covered anywhere: min/max written as floatstrings where min > max must be caught by a numeric comparison (base/job_templates/3.3.1--amount-min-greater-than-max.invalid.yaml uses numeric literals only). Same lexical-vs-numeric class as the INT bounds fixtures. Lower severity than the FLOAT parameter-definition gap, but currently a blind spot. Suggest a min/max floatstring min > max invalid fixture.
Contributor
Author
There was a problem hiding this comment.
Agreed, I will create more tests to cover this as well.
|
Also windows is failing |
leongdl
enabled auto-merge (squash)
August 28, 2026 22:13
leongdl
commented
Aug 28, 2026
| # https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md?plain=1#L1014 | ||
| # §3.3.2.2 L1059 `<AttributeCapabilityValue>` constraints apply AFTER the format string resolves | ||
| # https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md?plain=1#L1059 | ||
| template: |
Contributor
Author
There was a problem hiding this comment.
A [Format String](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md#73-format-strings) subject to the following constraints:
Minimum length: 1 character.
After the format string has been resolved:
Max length: 100 characters
Unicode alphanumeric characters in the latin character set, plus the underscore and hyphen characters.
Must start with either a letter character or the underscore character.
leongdl
commented
Aug 28, 2026
| parameterDefinitions: | ||
| - name: Feature | ||
| type: STRING | ||
| default: "not valid!" |
Contributor
Author
There was a problem hiding this comment.
A [Format String](https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md#73-format-strings) subject to the following constraints:
Minimum length: 1 character.
After the format string has been resolved:
Max length: 100 characters
Unicode alphanumeric characters in the latin character set, plus the underscore and hyphen characters.
Must start with either a letter character or the underscore character.
leongdl
commented
Aug 28, 2026
| taskParameterDefinitions: | ||
| - name: Frame | ||
| type: INT | ||
| range: ['1', '02', '003'] |
Contributor
Author
There was a problem hiding this comment.
leongdl
commented
Aug 28, 2026
| taskParameterDefinitions: | ||
| - name: Weight | ||
| type: FLOAT | ||
| range: ['1.5', '02.50'] |
Contributor
Author
There was a problem hiding this comment.
leongdl
commented
Aug 28, 2026
| # https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md?plain=1#L957 | ||
| # §3.3.1 L960 max may be a format string under FEATURE_BUNDLE_1, so the bound defers past decode | ||
| # https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/wiki/2023-09-Template-Schemas.md?plain=1#L960 | ||
| template: |
Contributor
Author
There was a problem hiding this comment.
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Conformance fixtures for every OpenJD 2023-09 field where the raw template and the resolved job can hold different values. Seventeen fixtures added and one strengthened, across 21 changed files. Eleven of the new fixtures plus the strengthened one are live and pass both reference implementations; six sit under
proposed/because they fail one implementation or await a spec ruling.The suite covers these fields well as literals and barely at all as format strings or string-form numerics. That asymmetry is the whole gap.
Why it matters
Not hypothetical. A service reading
chunks.defaultTaskCountout of the pre-resolution template rejected valid task-chunking jobs in production across three regions, because it saw the literal{{Param.ChunkSize}}where the resolved job carries anint. The template was valid and the resolved value was already available; the consumer read the wrong one.Writing the fixtures found four more defects that nothing in the suite could have caught.
Two coercion layers, and the seam between them
A field annotated
@fmtstringcarries one coercion. A parameter declared<integer> | <intstring>carries another. Chained, they are the real customer shape and the untested one.Every fixture whose subject is a
@fmtstringfield therefore writes its parameter value in the string form.chunks.defaultTaskCountresolves fromdefault: '03', so the asserted boundaries1-3/4-6/7-8hold only if the string parsed to 3 and the format string then resolved to it.notifyPeriodInSecondstakes a submitted'047', chaining three coercions with the reflection variable. Leading zeros are deliberate: a bare'3'is indistinguishable from a passed-through literal, and the fixtures forbid the un-normalized text.The assertions are identical to the native-integer form, so this is strictly stronger at no cost.
The
<intstring>and<floatstring>class had no job coverage at allJobIntParameterDefinitionandJobFloatParameterDefinitiondeclaredefault,allowedValues,minValueandmaxValueas a union with a string form, in base 2023-09, with no extension gate.IntRangeListandFloatRangeListdo the same for their elements.The suite had template-level accept coverage and no job-level coverage of any string-form definition field. Across 2,417
IntRangeListelements and 1,044FloatRangeListelements, not one used the string form.The bound case is the sharp one: when
minValueis the string'010'and the submitted value is the number50, the comparison must be numeric. A lexical comparison orders them differently, and nothing was checking.Fields no CLI prints
cancelation.notifyPeriodInSecondsas a format string was covered only by a fixture asserting the constantNOTIFY_OK, because neitheropenjd runnoropenjd summarysurfaces the field.WrappedAction.Cancelation.NotifyPeriodInSecondsfrom RFC 0008 closes it properly: the reflection variable carries the effective value after resolution, so a wrap script reads back what the job actually got.The resolved job
nameis similar. Both CLIs print it asRunning job '<name>'andJob: <name>, so it is assertable at base level without EXPR. Existing base fixtures assert a parallel substitution into a task argument instead, which proves the parameter resolved somewhere and says nothing about the name field.Live fixtures, all passing both CLIs
TASK_CHUNKING/jobs/default-task-count-format-stringdefaultTaskCountfrom'03'yields boundaries1-3,4-6,7-8; forbidsCHUNK:1-8TASK_CHUNKING/jobs/target-runtime-seconds-format-stringtargetRuntimeSecondsfrom'00', where 3.4.1.5 fixes scheduler behaviour so boundaries are deterministicWRAP_ACTIONS/jobs/wrap-cancelation-notify-period-fmtstring-resolvednotifyPeriodInSecondsread back through reflection; forbids the default and the unresolved literalbase/jobs/1.1.1--resolved-job-name-valuenamefield itself; forbids the default-derived namebase/jobs/1.1--path-default-joined-with-template-dirbase/jobs/2.3--int-param-intstring-default-resolvesdefault: '007'resolves to7; forbidsCOUNT:007base/jobs/1.1--int-intstring-bounds-satisfiedand--violation.invalidminValue/maxValuecompared numerically against a native submitted valuebase/jobs/1.1--int-intstring-allowedvalues-violation.invalidallowedValuesmembership decided numericallyFEATURE_BUNDLE_1/jobs/3.3.1--amount-min-format-string-resolves-to-non-number.invalidminmust be numericFEATURE_BUNDLE_1/jobs/3.3.1--amount-max-format-string-resolves-to-non-number.invalidmax, which is a different spec typeOne existing fixture strengthened
base/jobs/1.1--validation-valid-path-default-within-template-dir.test.yamlasserted the substring/outputagainst a default of./output./outputis a substring of./output, so an implementation that skipped the template-directory join passed it. I verified that by running it with the task hardcoded to print the un-joined value: it passed. Twoforbiddenentries close it, and the same mutant now fails.Proposed fixtures: four implementation defects and one spec question
Under
proposed/, which the runner's non-recursive glob skips, so the suite stays green. Promotion isgit mvup one directory with no edit. Each directory carries a README with the construct, the observed behaviour per implementation, and a classification.TASK_CHUNKING/.../default-task-count-...-resolves-to-zero.invalidpasses validation checks, runs, logsFrame(CHUNK[INT]) = 1-1FEATURE_BUNDLE_1/.../amount-max-...-resolves-to-zero.invalidmaxis<positivefloat>; Python rejects0, Rust runs. The only assertion distinguishingmaxfromminbase/.../format-string-in-anyof-...-invalid-value.invalidbase/.../format-string-in-allof-...-invalid-value.invalidbase/.../int-range-intstring-elements-normalizedand the float pairThe first four are one defect shape: a constraint that cannot be checked at decode because the value is a format string, deferred correctly, then never re-checked after resolution. Reviewers may prefer to treat them as one issue.
#161establishes the class with1.1.1--resolved-name-129-chars.invalid.test.yaml, whose comment notes "the literal is short, so only a post-resolution check catches this".The two range-element fixtures carry a spec question, not a defect.
<intstring>and<floatstring>are defined only as base-10 string representations, with nothing said about normalization, so['02']yielding2and yielding02are both defensible. The implementations disagree in opposite directions on different fields: for range elements Python preserves the literal and Rust normalizes; for a FLOAT parameter default they swap sides. AndEXPR/jobs/expr1.3.4--float-passthrough.test.yamlalready pins the verbatim reading for a default, so if the spec rules that<floatstring>normalizes, that landed fixture and mine cannot both be right. These two should not merge until that is settled.Verification
Every fixture run against both the Python and Rust CLIs, then mutation-checked: change the resolved value, leave the expectations, confirm it fails. Thirteen mutants, thirteen caught. The notifyPeriodInSeconds and job-name fixtures were additionally mutated by removing their parameter overrides, so a fixture satisfied by a template default rather than a resolved submitted value fails.
'03'to'abc'fails at parse rather than passing quietly.Slice results on this branch, both CLIs:
TASK_CHUNKING20/0,FEATURE_BUNDLE_156/0,WRAP_ACTIONS73/0,base661 passed with one pre-existing failure.proposed/confirmed undiscovered in every slice.CI is green on five of six legs. The Python CLI on Windows fails 13 fixtures, and that set is byte-identical to mainline's, so it is the known tolerated set rather than drift from this change. Two fixtures are POSIX-gated and log an explicit skip on Windows.
Locally I also saw
base/job_templates/3.5--env-script-onexit-only.yamlfail against my checked-out Rust CLI, but CI passes it, so that was version drift in my binary rather than a suite problem.Two defects in the suite's own machinery, not fixed here
Worth separate issues.
A non-
.invalid..test.yamlwith noexpected:block never inspects the return code inrun_job, so a test whose run errored is reported as passing. Proven with a template referencing an undefined parameter that Rust rejects outright.Nothing enforces that
proposed/stays undiscovered. Several PRs of known-failing fixtures now depend on a non-recursiveglobthat no test protects; making it recursive would silently turn the suite red.Not covered, and why
Positive assertions on resolved host-requirement values. No CLI surfaces them, and the eight WRAP_ACTIONS reflection variables are scoped to
<Action>fields, so reflection cannot reach them. The negative cases here are the reachable half.Capability
nameas a format string. The spec declares<AmountCapabilityName>"A string" at 3.3.1.1 and does not annotate it@fmtstring, yet Python resolves and re-validates it while Rust rejects the template at decode. A fixture today would pin an implementation rather than the spec, so this needs a ruling first. I would raise it as a spec question.Relationship to other open PRs
No file-level conflict: no open PR adds or modifies any path here, and none touches
TASK_CHUNKING/at all or adds ahostRequirements.amountsblock anywhere.#174introduces a self-asserting convention for single-taskjobs/fixtures. These are written against mainline with plainexpected.output; whichever merges second wants an instrumentation pass, not a merge resolution. Several fixtures here are multi-task, which#174excludes from self-assertion anyway.Separately for
#174: its rewrites of3.6--let-in-host-requirementsand7.3--param-in-capabilitymake both self-asserting while leaving their subjects unasserted. The first still asserts a parallelprintof the let binding rather than the resolvedamounts[].min; the second still has nohostRequirementsblock at all.