A residual-connection replacement that keeps multiple hidden-state streams and learns how each block reads from and writes back to them.
Normal Transformer blocks have one residual stream:
x = x + block(norm(x))
Hyper-Connections expand this into n parallel residual streams. Each attention or MLP branch reads a learned mix of the streams, runs the normal branch, then writes the output back into the streams. So the residual path becomes a small learned routing system instead of a single add.
In BlaGPT this is implemented as Dynamic Hyper-Connections with 4 streams, DHCx4.
For each branch, Hyper-Connections use three learned mixing objects:
A_m: mixes the residual streams into the branch inputA_r: carries and mixes the old streams forwardB: decides how strongly the branch output is injected into each stream
Static form:
branch_in = sum_s A_m[s] * stream[s]
branch_out = branch(norm(branch_in))
next_stream[u] = sum_s A_r[s, u] * stream[s] + B[u] * branch_out
With dynamic Hyper-Connections, A_m, A_r, and B also get token-dependent deltas predicted from the current streams:
A_m = A_m_static + delta_m(streams)
A_r = A_r_static + delta_r(streams)
B = B_static + delta_b(streams)
The deltas are zero-initialized and scaled by small learned factors, so training starts close to the static schedule instead of immediately scrambling the residual paths.
The implementation lives in bla_gpt/bla_gpt.py:
HyperConnection: stream mixing and dynamic projectionsHyperBlock: wraps the normal attention and MLP branches with Hyper-ConnectionsGPT.forward: expands the hidden state to[batch, seq, streams, dim]before the blocks and sums streams back to[batch, seq, dim]before the final norm
Configuration fields:
use_hyper_connections = True
hyper_num_streams = 4
hyper_dynamic = True
hyper_tanh = TrueThe registered experiment model is in bla_gpt/model_registers.py:
config.use_hyper_connections = True
config.hyper_num_streams = 4
config.hyper_dynamic = TrueTwo implementation details mattered:
- The dynamic projection weights are initialized to zero, with small
1e-2scales for the deltas. - Residual projection init is scaled by
sqrt(hyper_num_streams)when Hyper-Connections are enabled, because the model later sums multiple streams.
Run: best_hyper_connections_scale_fix
Final result on the best-model stack:
- step:
5100/5100 - validation loss:
3.2741 - train loss:
3.2388 - peak memory:
68729 MiB - step average:
928.26ms
This did not beat the old best around 3.2327. It also made training much heavier. The run saturated 8 H100 80GB GPUs and used roughly 69GB peak memory per GPU.
Hyper-Connections are neat, but in this BlaGPT run they were too expensive. The 4-stream dynamic version pushed memory and step time way up, while validation loss got worse than the previous best. For now it stays as an implemented experiment, not part of the best config.
Paper: Hyper-Connections
Implementation: bla_gpt/bla_gpt.py::HyperConnection, bla_gpt/bla_gpt.py::HyperBlock