Skip to content

Update pipeline.md following 19.0 propertizing #1495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eggrobin merged 2 commits into from
Aug 7, 2026
+8 −29
Merged

Update pipeline.md following 19.0 propertizing #1495

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2b20288
Update pipeline.md following 19.0 propertizing
eggrobin Aug 6, 2026
28f76a1
More documentation of cross-repository surgery
eggrobin Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 8 additions & 29 deletions docs/pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,14 @@ Prerequisites: proposal posted to L2, SAH agreed to recommend for provisional as
- [ ] Commit
- [ ] UTC decision — Check counts, code points, names, properties
- [ ] SAH report — Check counts, code points, names, properties
- [ ] Ken’s UnicodeData draft — [Check consistent](#ken-unicodedata)

---
If the proposal supplies LineBreak.txt:
- [ ] LineBreak.txt — Prepend lines from proposal
- [ ] Commit

If the proposal does not supply LineBreak.txt:
- [ ] LineBreak.txt — [Regenerate](#regenerate-linebreak) [TODO(markus): This should become « invoke Ken’s tool »]
- [ ] LineBreak.txt — [Regenerate](#regenerate-linebreak)
- [ ] Update modified lines
- [ ] Commit

Expand Down Expand Up @@ -63,7 +62,7 @@ reserved:

---
- [ ] In unicodetools/src/main/resources/org/unicode/text/UCD/AdditionComparisons,
copy template.txt to [RMG issue number].txt.
copy template.txt to [RMG issue number].txt, or sew-[SEW issue number].txt, whichever one is in the pipeline dashboard.
- [ ] Comparison tests — Write
- Examples:
- [straightforward characters](https://github.com/unicode-org/unicodetools/blob/08748760e371d9dbdc6a0fc883c68dff944648e2/unicodetools/src/main/resources/org/unicode/text/UCD/AdditionComparisons/182.txt#L11-L18),
Expand All @@ -84,7 +83,7 @@ PR preparation:
- [ ] Working group — Mention:
- Proposals from SAH — Link SAH issue
- Proposals from ESC or CJK — Mention ESC or CJK in the PR description
- [ ] RMG issue — Link
- [ ] RMG issue, if any — Link
- [ ] data-for-new — Set label
- [ ] pipeline-* — Set label:
- **pipeline-recommended-to-UTC** if the characters are not yet in the pipeline,
Expand Down Expand Up @@ -116,35 +115,15 @@ PR preparation:
- For proposals from CJK, file a PAG issue of type `Document`, citing the proposal.
Put the review in the `Background information / discussion` section, and link the pull request
in the `Internal` section. See, _e.g._, https://github.com/unicode-org/properties/issues/366.
- When a single SAH issue results in multiple PRs (e.g., because of separate UTC approvals), file
PAG issues for each subsequent PR, and link the PAG issue from the PR description.
- [ ] PAG dashboard status of SAH or PAG issue — Set to `Review`
- [ ] Pipeline dashboard PAG status of RMG issue — Set to `data review`
## Scripts

There are a variety of setups for unicodetools, depending on OS, in-source vs. out-of-source, git practices, etc.
If you take part in UCD development, feel free to add your own.

### Ken UnicodeData

Ken's files come from [here](https://corp.unicode.org/~book/incoming/kenfiles/) (select appropriate ucd version e.g. `ucd160` for Unicode 16.0). NOTE: this check is probably not applicable for `pipeline-provisionally-assigned` data where Ken does not yet have a draft.

eggrobin (Windows, in-source; the remote corresponding to unicode-org is called la-vache, Ken’s files are downloaded next to the unicodetools repository).

```powershell
$latestKenFile = (ls ..\UnicodeData-*.txt | sort LastWriteTime)[-1]
$kenUnicodeData = (Get-Content $latestKenFile)
git diff la-vache/main */UnicodeData.txt |
sls ^\+[0-9A-F] |
% {
$headLine = $_.line.Substring(1)
if (-not $kenUnicodeData.Contains($headLine)) {
$codepoint = $headLine.Split(";")[0];
echo "Mismatch for U+$codepoint";
echo "HEAD : $headLine";
echo "Ken : $($kenUnicodeData.Where({$_.Split(";")[0] -eq $codepoint}))";
}
}
```

### Merge

eggrobin (Windows, in-source; the remote corresponding to unicode-org is called la-vache).
Expand Down Expand Up @@ -233,8 +212,8 @@ git commit -m GenerateEnums


### Run comparison tests
eggrobin (Windows, in-source; replace $RMG_ISSUE by the RMG issue number, or define it as that number).
eggrobin (Windows, in-source; replace $RMG_ISSUE by the RMG issue number or sew-[SEW issue number], or define it as that).
```powershell
mvn test -am -pl unicodetools "-DCLDR_DIR=$(gl|split-path -parent)\cldr\" "-DUNICODETOOLS_GEN_DIR=$(gl|split-path -parent)\unicodetools\Generated\" "-DUNICODETOOLS_REPO_DIR=$(gl|split-path -parent)\unicodetools\" "-Dtest=TestTestUnicodeInvariants#testAdditionComparisons" -Dsurefire.failIfNoSpecifiedTests=false -DtrimStackTrace=false "-DRMG_ISSUE=$RMG_ISSUE"
mvn test -am -pl unicodetools "-DCLDR_DIR=$(gl|split-path -parent)\cldr\" "-DUNICODETOOLS_GEN_DIR=$(gl|split-path -parent)\unicodetools\Generated\" "-DUNICODETOOLS_REPO_DIR=$(gl|split-path -parent)\unicodetools\" "-Dtest=TestTestUnicodeInvariants#testAdditionComparisons" "-Dsurefire.failIfNoSpecifiedTests=false" -DtrimStackTrace=false "-DRMG_ISSUE=$RMG_ISSUE"
```
Results are in Generated\UnicodeTestResults-addition-comparisons-[RMG issue number].html.
Results are in Generated\UnicodeTestResults-addition-comparisons-$RMG_ISSUE.html.
Loading