Obscure, too terse messages from GPD

[pmilford]

HI,
I'm working on a problem, the number of phases, experimental configurations has grown a lot. I'm finding
too many of the intermediate messages/communications as too terse and difficult to review the meanings...

See below for an example, even though I have been working on this project, I have a lot of trouble understanding the issue
and the discussion around it...
What is dose=B, I need to look that configuration up somewhere. (it is actually a particular test configuration that GPD uses the
term dose, rather than say experimental configuration. or config1(L1, L2 only)). It appeared switch to this terminology about 1/2 way
through the phases...
Where is the original grad_clip=1.0 set and why?

I'm happy with the apparent rigor, not the too terse description and summary of the state.

Peter

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GPD ► PHASE 33 — WAVE 1 HALTED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Plan 33-01 halted at the first Stage A cell. Wave 2 (33-02) is blocked.

Trigger: Cell B (dose=B, seed 330015) completed finite (val_ppl_final=9.42, 45.9s) but 95.8% of post-warmup steps were clipped (median pre-clip grad-norm 2.95, max 13.10) against the frozen
grad_clip=1.0. Primary halt rule fraction_clipped_steps > 0.5 on dose=B fired. This is Risk 8b from RESEARCH materializing: the Phase 32 2M gradient-norm distribution does not transport to
5M.

Per plan discipline, the executor did NOT retune — altering grad_clip, lr, or recipe is fp-optimizer-swap territory and requires user decision.

State:

• ✓ Tasks 1–3 complete (config, dose sets, runner) — all committed
• ◆ Task 4: 1/6 Stage A cells (only B executed)
• ○ Tasks 5–6 skipped
• ✓ Task 7 halt branch executed; HALT_CHECKPOINT.flag sentinel committed
• ✗ plan_33_02_handoff.json deliberately not written
• DEVIATION-33-01-01 recorded

[SergioHC95]

Peter, thanks for sharing! Very long-horizon projects are indeed a complex case, and we are working on improving performance of GPD there in various ways (carrying knowledge templates, tracking progress checkpoints, performing more frequent validation cross-checks, locking more general conventions, etc).

If you are willing to share some of your conversation logs with GPD, that would be very helpful for us to identify exactly where GPD derailed and how to prevent it (of course, no pressure!). Equally useful if you are able to pinpoint this more specifically. Looking forward to helping!

Thanks!