v.ast, v.checker: fix infinite loops and segfaults in generic instantiation and cyclic aliases

[tailsmails]

What this does

Fixes the compiler hangs (100% CPU forever) and segfaults (stack overflows) when the checker hits recursive generics or cyclic aliases. Repros for all 5 cases are in the comments below.

The idea is simple: put cutoff limits on the code paths that can loop forever (rustc and GHC do the same thing), and fix the validation so a generic sum type can't sneak itself in as a variant.

What changed

• fully_unaliased_type can't spin forever on circular aliases any more (100 unwrap iterations max).
• unwrap_generic_type_ex_with_depth tracks recursion depth and panics cleanly at 256, instead of overflowing the stack. The depth is passed as a local arg - the earlier version with counters on Table hit a data race on the Linux Clang CI.
• the generic fn post-processing loop bails with a proper error after 128 rechecks, instead of hanging. Note this also replaces the hardcoded 1_000_000 const that master currently has there.
• sum_type_decl strips the generic brackets before comparing names, so type MySum[T] = T | MySum[MySum[T]] is rejected as circular like it should be. The comparison is module-aware now, so a same-named type from another module doesn't produce a false positive.
• in generic_insts_to_concrete, hitting a circular sum type now skips just that one instantiation instead of returning from the whole function, which was silently dropping every instantiation after it.

The limits are flags now

These started out as hardcoded consts. After the review feedback I just exposed them as flags instead of adding real cycle detection - detection would tax some of the hottest checker paths to catch code that (honestly) nobody writes by hand, and anyone who genuinely needs a bigger limit can pass it:

• -generic-fn-inst-limit (default 4096)
• -generic-inst-name-len-limit (default 8192)
• -generic-inst-depth-limit (default 256, clamped to 512 - go deeper and the native stack dies before the guard can fire, I ran into that segfault while testing at 512)
• -alias-unwrap-depth-limit (default 100)
• -generic-fn-postprocess-iters (default 128)
• -max-postprocess-iterations (default 100000)

Defaults are the same values the consts had, so nothing changes if you don't pass the flags. Panic/error messages tell you which flag to raise. Also documented in v help build.

Tests

vlib/v/tests/recursion_cutoff_flags_test.v - every crash repro now exits with a controlled error/panic instead of hanging or segfaulting, each flag demonstrably changes its limit, the 512 clamp works, and valid code still compiles and runs untouched with the defaults. 29 asserts total.

[tailsmails]

bug1.v (Recursive generic functions infinite loop)

fn foo[T]() { foo[[]T]() }
fn main() { foo[int]() }

bug2.v (Nested generic structs stack overflow / segfault)

struct Box[T] { Box[Box[T]] }
fn main() { b := Box[int]{} }

bug3.v (Nested generic methods infinite loop)

struct Box[T] { val T }
fn (b Box[T]) foo() { Box[Box[T]]{}.foo() }
fn main() { Box[int]{}.foo() }

bug4.v (Circular reference bypass in generic sum types infinite loop)

type MySum[T] = T | MySum[MySum[T]]
fn main() { mut x := MySum[int]{} }

---

Expected Outputs with these patches (No hangs, no segfaults):

For bug1.v and bug3.v (Graceful loop termination):

test.v:1:1: error: generic function post processing reached the cutoff limit of 50 iterations, probably due to an infinite generic instantiation loop
    1 | fn foo[T]() { foo[[]T]() }
      | ~~~~~~~~~~~
    2 | fn main() { foo[int]() }

For bug2.v (Controlled limit panic instead of Segmentation Fault):

V panic: generic instantiation depth limit exceeded

For bug4.v (Consistent circular sum-type verification):

test.v:1:21: error: sum type cannot hold itself
    1 | type MySum[T] = T | MySum[MySum[T]]
      |                     ~~~~~~~~~~~~~~~

[tailsmails]

I have also found a 5th related segmentation fault in convert_generic_type with this recursive alias pattern, and I will address it in a separate PR soon..

[tailsmails]

Update: Found another related segfault while testing this branch.

It happens when resolving recursive generic function pointer aliases like this:

type Func[T] = fn (f Func[[]T]) T

fn main() {
	_ := Func[int](none)
}

On master, this crashes with a segmentation fault in v__ast__Table_convert_generic_type or unwrap_generic_type_ex_with_depth.

What was going wrong:
The mutual recursion between convert_generic_type and unwrap_generic_type_ex_with_depth bypassed the local depth_guard arrays. Since they call intermediate helpers (which reset the local depth arguments back to 0 or []string{}), the local limit checks were never triggered.

The fix:
To solve this cleanly, I added two dynamic counters to the Table struct (convert_generic_type_depth and unwrap_generic_type_depth). Tracking the recursion depth directly on the mutable Table works perfectly across mutual recursion boundaries. The compiler now safely catches the infinite recursion and panics with generic instantiation depth limit exceeded instead of crashing.

[medvednikov]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab9edcefea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Replace the total generic-instantiation cap with cycle detection

When a valid program specializes one generic function for 257 distinct concrete type lists, the map entry reaches length 257 and every subsequent call to register_fn_concrete_types—even a duplicate that would otherwise return false—panics here. The number of finite specializations is not evidence of recursive expansion, and generated serializers or dispatch code can legitimately exceed this threshold, so this turns valid compilation into a compiler crash; guard actual recursive growth instead of the accumulated specialization count.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Allow finite generic instance names longer than 1 KiB

A finite generic instantiation can exceed 1,024 bytes without containing a cycle—for example, a generated generic with many concrete parameters or nested qualified type names—yet it now unconditionally panics here. Because V does not otherwise impose this identifier-length restriction, valid source that previously compiled now crashes; recursion should be bounded by the instantiation graph rather than the rendered name length.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve finite generic chains beyond 50 rechecks

A valid chain of more than roughly 50 generic functions can require more than 50 passes when declarations are ordered from the innermost function to the outermost and only the outermost function is initially instantiated: each pass discovers the concrete type for a function that was already visited earlier in that pass. This fixed cutoff therefore emits an infinite-instantiation error for a finite chain that previously converged; use repeated-state/no-progress detection or a limit that does not reject ordinary finite propagation.

Useful? React with 👍 / 👎.

[tailsmails]

I tweaked the constants a bit.

• Kept the depth limits at 256 and 100. I tried bumping them (e.g., to 512),
  but it just causes a segfault before V can even step in and panic.
• Bumped the size/count limits (generic_fn_inst to 4096, name_len to 8192, and iterations to 100k).
  Honestly, it's highly unlikely most developers will ever hit these numbers in real-world projects, but this safely covers the theoretical edge cases mentioned by Codex without causing any harm.

[tailsmails]

Raise generic post-process limit to 256

The old limit of 50 was too tight for complex nested generics, leading to false-positive loop errors on valid code. At the same time, we can't set it too high because actual infinite loops will hang the compiler and spike CPU usage.
Setting the cutoff to 256 is a safe middle ground. It prevents false positives for deep generic chains without adding any complex runtime overhead, and ensures the compiler still fails fast if a real loop happens.

[tailsmails]

Raise generic post-process limit to 256

The old limit of 50 was too tight for complex nested generics, leading to false-positive loop errors on valid code. At the same time, we can't set it too high because actual infinite loops will hang the compiler and spike CPU usage. Setting the cutoff to 256 is a safe middle ground. It prevents false positives for deep generic chains without adding any complex runtime overhead, and ensures the compiler still fails fast if a real loop happens.

I think maybe 128 is safer than 256

[tailsmails]

Hi there. All other OS checks ran smoothly, but Linux Clang picked up a data race in Table. Since those depth counters on Table are shared across threads, they caused concurrent writes.
I've just updated the PR to pass the depth counter as a local argument to make it thread-safe.

[tailsmails]

Everything is going well so far. We just need to wait for the Actions to complete.

[tailsmails]

All the main checks are passing now.

The cutoff limits are still hardcoded consts for the moment and I plan to expose them as compiler flags later (e.g. -generic_inst_depth=512 or etc.)
so users can adjust them as needed.
For now, though, priority is on fixing the root cause of the bug properly, and keeping things stable.

Happy to change direction if you have other thoughts!

Update: the cutoff limits are now compiler flags, not hardcoded consts: -generic-fn-inst-limit, -generic-inst-name-len-limit, -generic-inst-depth-limit (≤512 clamped), -alias-unwrap-depth-limit, -generic-fn-postprocess-iters, -max-postprocess-iterations. Defaults match the previous consts, so behavior without flags is unchanged. Panic/error messages name the flag to use. Covered by vlib/v/tests/recursion_cutoff_flags_test.v (29 assertions).

[tailsmails]

I'm working on a few new features on top of this; I'll publish them once they're ready.

[tailsmails]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[tailsmails]

To use Codex here, create a Codex account and connect to github.

bruh

[tailsmails]

Ok, now we need to wait for the Actions to complete again.