[pbednarcik]

Builds on #623 (TextMateSharp 2.0.5); the new change is the second commit. It also changes TokenizeViewPortCore, as
#622 does, so whichever lands second needs a small rebase. #622's tests change with it: one that counts on the forced
first line goes, one detects the pass another way, and the harness lays out after pumping, because the pass now
redraws when it ends. Merged that way, all 588 tests pass.

Summary

On every scroll-offset change, TokenizeViewPortCore calls TMModel.ForceTokenization for all visible lines at once on
the UI thread. With a slow grammar, the frame after a far jump and any input behind it wait for the whole viewport: on a
210,000-line (9 MB) SQL file the frame took 95-113 ms at p95, and an input job queued with the jump waited 121-196 ms
at p95.

Change

The pass tokenizes only the visible lines that are invalid or have no tokens, in runs of up to 8 lines on the UI thread.
After a run, if 4 ms have passed, the rest continues in slices posted at DispatcherPriority.Background, below input
and rendering. A newer viewport replaces the remaining range, Dispose stops the pass at the next run, and an exception
that escapes a run ends it. An edit up to the viewport's last line ends the pass, which starts again from the
viewport of the next layout: the view keeps its scroll offset, so the edit moves lines across the pass's line
numbers. An edit below the viewport leaves the pass running. The 4 ms is when the next run may start, not a limit on
how long one takes.

The pass redraws the lines it tokenized once: when it ends, when a newer viewport replaces it, or when the document
starts changing. A redraw per run rebuilt the viewport's visual lines between slices. The coloring transformer leaves
to that redraw only the token events a run raises on the UI thread for its own lines while the document is
unchanged and the model not disposed; events from TMModel's thread, and for any line outside the current run, redraw
as before. Disposing the model posts a redraw of the whole view, so lines the pass tokenized and had not yet redrawn
are rebuilt, also when a line transformer disposes it during layout. A line transformer that pumps the dispatcher
during layout runs posted slices and redraws inside it, where redrawing a line being built throws. A redraw the model
asks for while the view builds its lines, or before the model has seen a build end, redraws the whole view at the
start of the next build, which VisualLineConstructionStarting announces before any line is reused. That waiting
redraw holds only the view, not the exception handler, so a disposed model and its document are not kept alive by it,
and a view that measures with no document drops it. It runs inside that build, so what it throws goes to the layout, as
a line transformer's exception does. The coloring transformer's own posted redraws are as in #623.

Skipping lines that look valid needs TextMateSharp 2.0.5: before it, TMModel could leave a line marked valid with tokens
of its old text or old start state (danipen/TextMateSharp#137 and danipen/TextMateSharp#139).

On the SQL file the far-jump frame p95 becomes 28.5-37 ms and the input wait p95 28.5-41 ms. The time until every visible
line has tokens is p95 95-109.5 ms on #623 and 92-119 ms with this PR. A scroll on a settled file now tokenizes nothing,
where the forced first line made TMModel walk to the end of the file (danipen/TextMateSharp#142 bounds that walk): a
3-line wheel stream at 60 Hz kept the UI thread busy 45-54% before and 5-11% after. Typing that scrolls the view
sideways rebuilds fewer lines: 300 keys on a generated 190,000-line C# file built 6,571-6,640 visual lines before and
3,164-3,196 after.

Cost

• Dragging the scrollbar through a file that is still being tokenized: frames stay smooth (interval p95 73-81 →
  23-27 ms on SQL), but lines are mostly plain until release, because each scroll event starts a new pass that
  overtakes the last one. Frames in full colour: 70-76% → 2-4% on SQL, 68-83% → 2-12% on C#. Once the file is
  tokenized: 89-98% → 99.9%.
• Dragging over the settled SQL file: the longest frame is 32-47 ms, against 24.5-38 ms on Bump TextMateSharp to 2.0.5 #623.
• Lines that look valid but are stale are left to TMModel's thread. That happens in two cases. One is a line whose
  start state an edit above just moved; it updates when TMModel's thread reaches it, as it does on Bump TextMateSharp to 2.0.5 #623 without a
  scroll. The other is a line the grammar threw on; Bump TextMateSharp to 2.0.5 #623's forced pass repaired it on the next scroll if the grammar
  didn't throw again, and the fix belongs in TMModel's catch.
• A UI thread kept busy above Background priority holds the slices back.

Not covered

Measured on one Windows machine; macOS and Linux are not measured. 4 ms and 8 lines are the best of the settings tried
on one slow and one fast grammar, on TextMateSharp 2.0.4, and were not tuned again on 2.0.5.

Tests

TokenizeViewPortCore can return before every visible line has tokens and redraws when the pass ends, so headless tests
that scroll and then read tokens or visual lines pump the dispatcher and lay out.

New TextEditorModelViewPortTokenizationTests, 37 headless tests; TMModel's own thread is held where a test needs the
pass alone:

• the pass runs on the calling thread, its first slice stops after one 8-line run, and no run goes past the viewport;
• a newer viewport replaces the remaining range, also when a listener starts it during a run;
• the remaining slices run below a queued input job, and only one continuation is queued;
• valid lines are not tokenized again, a run takes in only the lines that need tokens, and lines a grammar change left
  without tokens are tokenized;
• after a run changes the next line's start state, the following lines end with tokens of the new state, also a line
  TMModel's thread was tokenizing from the old state (both fail on TextMateSharp 2.0.4);
• the lines a pass tokenized are redrawn once when it ends, when it is replaced and when the document changes, also
  after an earlier edit; a pass that tokenized nothing redraws nothing; tokens from another thread during a run, a line
  a listener forces above or below a run or after an edit during one, and a ForceTokenization outside a run, are
  still redrawn without waiting for the pass; disposing the model during a run, between slices or during layout
  redraws the lines the pass tokenized, and tokens a run stores after the model is disposed are redrawn;
• a dispose, or a pass that ends, in a dispatcher frame a line transformer pumps during layout throws nothing and
  redraws the lines the pass tokenized after the layout, also for a model created and disposed during that layout;
  a disposed model and its document are not kept alive by a view that then shows no document and is not measured
  again, also when the exception handler's owner holds them or the redraw after dispose throws through a handler that
  rethrows, and a measure with no document drops the waiting redraw; a redraw handed to the next build reports what
  its invalidation throws, once to a handler that rethrows it when a newer viewport starts that redraw;
• an edit up to the viewport's end, a deletion inside or above it or one a listener makes during a run, ends the
  pass, which then tokenizes the next layout's viewport, and only that layout starts a pass; an edit below the
  viewport leaves the pass running;
• Dispose stops the slices, also during a run; an exception that escapes a run ends the pass and still redraws the lines it tokenized when the handler rethrows, and one from a replaced request leaves the
  newer range pending.

All 576 tests pass in Release.

Measurements

Windows 11, .NET 10, 32 logical CPUs, 120 Hz display, a bench app on Avalonia 12.1.4. #623 against this PR, alternated,
two runs: far jumps 3 rounds a run (80 jumps to random lines after 15 warm-up jumps), the rest 2 rounds a run. Ranges
over all rounds, ms unless marked. Frame = until the second animation-frame callback after the jump; colour = until
every visible line has tokens; input wait = an Input job posted with the jump.

	#623	this PR
SQL far jump: frame p50 / p95 / max	32.5-37 / 95-113 / 140-170	12-15 / 28.5-37 / 36-65
SQL far jump: input wait p50 / p95 / max	39-45 / 121.5-196 / 192-409	9-13 / 28.5-41 / 46-73
SQL far jump: colour p50 / p95	31-37 / 95-109.5	30-40 / 92-119
190,000-line C#, far jump: frame p50 / p95	19-21 / 27-32	12-13 / 16-20.5
190,000-line C#, far jump: input wait p50 / p95	22-30 / 39-71	7.5-8.5 / 15-18
190,000-line C#, far jump: colour p50 / p95	19-21 / 27-32	19-20 / 27.5-30
12,860-line C#, far jump: frame p50 / p95	11-17 / 26.5-34	10-14 / 20.5-25
12,860-line C#, far jump: input wait p95 / colour p95	39-67 / 24-30.5	16-19 / 24-29
SQL far jump: GC pause per 80 jumps	119-129	119-127
3-line wheel at 60 Hz, settled: UI thread busy, SQL / C#	45-54% / 47-53%	5-11% / 6-9.5%
Same: frame interval p99, SQL / C#	14.5-17 / 14-17	13.5-14.5 / 10-15.5
Same: visual lines rebuilt / GC pause, SQL	1,866 / 121-128	1,151 / 66-70
Held scrollbar, settled: UI thread busy, SQL / C#	53-59% / 45-56%	17-20% / 18-21%
Same: frames in full colour / colour after release, SQL	89-95% / 10-13	99.9% / 19-33
Same: longest frame, SQL / C#	24.5-38 / 17.5-24	32-47 / 17-23
Held scrollbar, still tokenizing: frames in full colour, SQL / C#	70-76% / 68-83%	2-4% / 2-12%
Same: frame interval p95, SQL / C#	73-81 / 24	23-27 / 14-15
300 keys, sideways scrolling, C#: key frame p50 / p99 / GC pause	24.4-24.5 / 58 / 158-161	16.5-16.6 / 55.5-57 / 117-122
Same, SQL: visual lines rebuilt / key frame p99	4,007-4,668 / 24.5-27	3,244-3,330 / 24-25

The held scrollbar streams 233 lines an event at 60 Hz for 6 s. Frame interval is the time between animation-frame
callbacks. On the wheel stream the interval p95 splits into two groups across rounds, so p99 is the row to read. One
#623 round of the C# typing case failed in the bench's undo after typing (a NullReferenceException in TextArea's
input-method client), so that row's #623 figures are over three rounds.

[Copilot]

🟡 Changes recommended

A rethrown exception in a later tokenization slice skips the pending redraw, leaving previously processed lines visually stale until another redraw is triggered.

1 open issue

• Flush pending redraws when a slice handler throws · New

What changed in this PR

This PR reduces UI-thread blocking during TextMate viewport tokenization by splitting work into short runs and deferring remaining slices below input and rendering priority.

Changes:

• Tokenizes only visible lines needing tokens, using runs of up to 8 lines with a 4 ms slice budget.
• Batches redraws and handles viewport replacement, document edits, disposal, and layout reentrancy.
• Upgrades TextMateSharp to 2.0.5 and adds headless regression tests.

File	Description
test/​AvaloniaEdit.Tests/​TextMate/​TextEditorModelViewPortTokenizationTests.cs	Tests slicing, scheduling, redraws, disposal, and exception handling.
src/​AvaloniaEdit.TextMate/​TextMateColoringTransformer.cs	Defers redraws for token events owned by the active viewport run.
src/​AvaloniaEdit.TextMate/​TextEditorModel.cs	Implements time-sliced tokenization and coordinated redraw handling.
Directory.Packages.props	Updates TextMateSharp and its grammars package to 2.0.5.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Pending viewport tokenization retains pre-edit line indices after structural edits, which can skip surviving visible lines and leave them without tokens.

1 open issue

• Rebase pending tokenization range after line deletions · New

1 resolved since last review

• Flush pending redraws when a slice handler throws

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

A stale token run can suppress redraw events after an overlapping viewport request, leaving visible lines uncolored because suppression does not check the request version.

1 open issue

• Prevent stale runs from suppressing viewport redraws · New

1 resolved since last review

• Rebase pending tokenization range after line deletions

_{Review effort: Balanced}