pfsync (proto 240) never transmitted from one HA node — bulk transfer fails every time, live sync one-directional

[MikeBishop]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
• AI tools were used to create at least part of the text submitted herewith.

If AI was used, please disclose:

• Model used: Claude
• Extent of AI involvement: Proposed hypotheses and suggested debug steps; authored summary after debug session

Summary

In a two-node OPNsense CARP/pfsync HA pair, one node (opnsense-primary) never transmits a single pfsync frame, in either the initial bulk state-table transfer or ongoing incremental sync. This has been confirmed at four independent layers on the affected node — kernel counters, a capture on the physical interface, a capture on the underlying Proxmox host tap, and a capture on the pfsync0 pseudo-interface itself — all showing zero output.

The other node (opnsense-backup) sends pfsync normally — its outbound frames are confirmed reaching primary via matching netstat counters and packet-for-packet matching tcpdump captures on both ends. Backup's receive path, however, has never actually been observed working: netstat -s -p pfsync on backup shows 0 packets received (IPv4), total, since boot. Given there are only two nodes in this pair, that's fully consistent with — and not independent evidence against — primary's transmit failure: backup has nothing to receive because primary has never sent it anything. So only one direction of this link (backup → primary) has actually been demonstrated healthy; the reverse direction is unconfirmed, not confirmed-and-broken-elsewhere. The failure survives a full VM restart (not just an in-OS reboot) and is not explained by anything in the pf ruleset, HA config, ARP, ARP/FDB entries, EVPN control-plane state, or per-queue NIC driver statistics, all of which check out clean and symmetric between the two nodes.

Because the bulk transfer fails every time, CARP demotion never fully clears and accumulates a large stuck negative value on the master node, which in turn caused CARP Maintenance Mode to fail to force a failover (consistent with the existing report at opnsense/core#5930 re: maintenance mode not reliably working in virtualized environments) — a manual CARP disable was required to force mastership over to the healthy node.

Environment

• OPNsense version: OPNsense 26.1.11_10 (amd64)
• Platform: Proxmox VE, QEMU/KVM (meta: creation-qemu=11.0.2)
• Two single-socket, 8-core, 8GB VMs, one per physical Proxmox host (pve-a, pve-b)
• NIC model: virtio, queues=8 on every interface, including the pfsync-dedicated NIC
• pfsync interface: vtnet5 (OPNsense-internal name opt3), dedicated point-to-point /30 (169.254.10.0/30, primary=.1, backup=.2), MTU 8930
• The sync bridge on the Proxmox side is not a plain Linux bridge — it's a VXLAN/EVPN overlay (VNI 3) under a VRF, configured nolearning, with FRR handling MAC/IP route distribution via BGP EVPN
• pfsyncversion (config token) / protocol version: 1400 (PFSYNCv5 in captures)
• hasync config block matches exactly between nodes apart from the peer IP (verified via /conf/config.xml)

Symptom

• Master node (opnsense-primary) accumulates a large state table over time (thousands of states); backup accumulates only what it creates for its own local services (dozens).
• netstat -s -p pfsync on primary shows normal, climbing receive counters (states arriving from backup) but completely static send counters — state inserts sent, end of frame marks sent, etc. do not move at all across a full working session: multiple curl-triggered new states, a pfctl/HA config reapply, an ifconfig pfsync0 create re-parameterization, and a full restart of the VM.
• Every pfsync bulk transfer attempt (triggered at boot, at manual pfsync0 reconfiguration, and at HA config reapply) fails with carp: demoted by -240 to X (pfsync bulk fail) in dmesg. The interval between pfsync bulk start and pfsync bulk fail is consistently ~65–90 kernel ticks, observed identically across multiple independent trigger events on both the master role and — earlier in the pair's life, on the other node while it briefly held the requester role — meaning the ~65–90 tick timeout itself is not new or unique to this node; what's unique to this node is that it never actually gets a frame onto the wire in the first place, request or otherwise.

Reproduction

1. Bring up a two-node OPNsense 25.x HA/CARP pair on Proxmox VMs, one VM per physical host, connected via a dedicated pfsync interface on a VXLAN/EVPN-backed bridge.
2. Let the pair run under normal traffic for some time (state churn on both nodes).
3. On the node that will exhibit the bug (in our case, whichever node was master at initial CARP bring-up): netstat -s -p pfsync | grep "state inserts sent".
4. Trigger new state creation (e.g. curl to any external host).
5. Re-check the same counter. Expected: it increases. Observed: no change, ever, regardless of how the state was created or how much time passes.
6. Confirm nothing crosses the wire for proto 240 from this node's pfsync IP, at any of: the guest's physical interface (tcpdump -ni vtnet5 -n proto 240 and src host <this-node-ip>), the Proxmox host's tap device for that NIC (tcpdump -ni tapNNNiM proto 240), and the pfsync0 pseudo-interface itself (tcpdump -ni pfsync0 -n).
7. Force a fresh bulk attempt (ifconfig pfsync0 create syncdev <if> syncpeer <peer-ip> maxupd 128 up, or toggle/reapply HA settings) — dmesg logs a pfsync bulk start followed ~65–90 ticks later by pfsync bulk fail, with zero packets observed in any of the three captures above during that window.
8. Fully restart the VM (host-level qm stop/qm start, not an in-guest reboot) — the behavior is identical post-restart. Send counters remain frozen from first boot.

What's been ruled out

Working through this collaboratively (see note on process below), the following were checked and are not the cause:

• ARP — resolves correctly and is cached (arp -an shows a valid, current entry for the peer).
• Physical interface health — netstat -i shows 0 input/output errors on the pfsync NIC across millions of packets of other traffic.
• pfsync0 binding — ifconfig pfsync0 reports the correct syncdev, syncpeer, maxupd, and version, matching the saved config exactly (syncok: 1 throughout).
• HA/pfsync config file — <hasync> block in /conf/config.xml is symmetric between both nodes (same pfsyncinterface, pfsyncversion; peer IPs correctly cross-referenced).
• pf ruleset on the pfsync interface — no block/quick rule shadows the interface; all pass rules for that interface show normal, expected evaluation counts unrelated to pfsync.
• Multiqueue/RSS on the virtio NIC — per-queue sysctl dev.vtnet.N.txqM.opackets shows all 8 transmit queues actively moving packets for other traffic sharing the same physical interface (an rsync process, unrelated to pfsync), ruling out a stuck-queue/hash-collision explanation for why proto 240 specifically never appears.
• EVPN control plane / bridge FDB — bridge fdb show on both Proxmox hosts shows correct, mirrored extern_learn entries for each node's pfsync MAC pointing at the correct peer VTEP address; the overlay's control plane is healthy and symmetric.
• VM/NIC configuration drift — qm config for both VMs shows identical NIC model, queue count, and bridge attachment for the pfsync interface; the only structural difference is which physical Proxmox host each VM runs on.
• A one-off wedged runtime state — ruled out by the VM restart test; the exact same failure signature (frozen send counters, identical bulk-fail timing) reappeared from first boot.
• CARP role / demotion arithmetic causing an apparent "won't fail over" symptom — this part is expected behavior, not a bug: the affected node's stuck demotion value (advskew + demotion) was large enough to keep it winning CARP elections regardless of Maintenance Mode, consistent with Enter Persistent CARP Maintenance Mode doesn't do anything core#5930's notes on maintenance mode being unreliable in virtualized environments; a manual disable CARP was needed to force failover. This is a downstream consequence of the pfsync bug, not a separate issue.

Current understanding / ask

At this point the failure appears to be entirely inside the FreeBSD kernel's carp/pfsync send path on the affected node — something prevents pfsync_out/the carp bulk-request logic from ever constructing or queuing an outbound frame, upstream of the network stack, the driver, and everything below it. We don't have kernel-debugging tooling (DTrace/ktrace on the relevant kqueue functions) applied yet, and would appreciate guidance on:

• Any known FreeBSD/OPNsense issue matching "pfsync send counters frozen from boot, receive path unaffected, survives full VM restart"
• Whether there's a known interaction between pfsync's output path and any of: VXLAN/EVPN-backed bridges specifically (as opposed to a plain L2 bridge or physical NIC), the nolearning bridge flag on the underlying overlay, or virtio-net on recent FreeBSD/OPNsense releases
• What kernel-level tracing (ktrace, DTrace probes on carp/pfsync, or a debug sysctl) would help pin down where in the send path the frame is being dropped or never generated

Happy to gather ktrace/DTrace output, pfsync_undefer/related sysctl values, or anything else that would help narrow this further.

---

Note on how this report was produced

This write-up documents a diagnostic session conducted collaboratively between the system owner (who ran every command and supplied all raw output below) and Claude (Anthropic's AI assistant), which proposed hypotheses, suggested each diagnostic command, and interpreted results in real time. In the interest of full transparency:

• All commands were run by the human operator, on real production hardware, and all output quoted or summarized above is genuine, unedited system output.
• The AI proposed the diagnostic sequence and several intermediate theories, some of which were wrong and were later disproven by further testing — including, at various points, an incorrect read of packet-capture directionality, a since-abandoned theory that the OPNsense virtio NIC's multiqueue/RSS hashing was dropping proto 240 traffic onto a stalled queue, and an incorrect prediction that a full VM restart would resolve the issue. Each of these was corrected once contradicting evidence came in, and this report reflects only the conclusions that survived that process, not the discarded ones.
• The final conclusion — a primary-side, boot-persistent, kernel-internal pfsync transmit failure — is the product of that iterative process, not a single diagnosis. Anyone acting on this report should treat the ruled-out list as reasonably solid (each item was checked against direct command output) but should independently verify anything critical before relying on it.

[Monviech]

This analysis was generated collaboratively with OpenAI ChatGPT (GPT-5.6 Thinking). It represents an independent technical review of the report and should be treated as analysis rather than confirmation of a software defect.

Overall assessment

The report is very well documented, but it suffers from a classic diagnostic problem:

It has accumulated an enormous amount of evidence about the environment while skipping the one assumption that actually needs to be proven.

The report gradually transforms:

"I do not observe packets."

into:

"The FreeBSD kernel never constructs packets."

Those are not equivalent statements.

What the report actually proves

These are the observations that appear reasonably solid.

1. Backup → Primary traffic exists

This appears well demonstrated.

The backup node transmits pfsync packets, and the primary receives them. Multiple captures and counters apparently agree.

That direction appears healthy.

2. Primary apparently never transmits pfsync

This also appears well demonstrated.

Several independent observations agree:

• tcpdump on pfsync0
• tcpdump on the physical guest interface
• capture on the Proxmox tap interface
• pfsync protocol counters

All apparently show zero transmitted pfsync packets from the primary.

This is the strongest piece of evidence in the report.

3. Bulk synchronization fails

This also appears credible.

They repeatedly observe something equivalent to:

pfsync bulk start
...
pfsync bulk fail

with approximately the same timeout.

That is a real observation.

4. A full VM restart does not change the behavior

This is useful information.

It indicates that the problem is probably not merely a transient userspace process state or a temporarily wedged virtual NIC queue.

It does not, however, prove a kernel defect.

A VM restart preserves:

• the OPNsense configuration
• the PF ruleset generated from that configuration
• loader settings and tunables
• VM hardware settings
• interface assignments
• the host and overlay environment

Therefore, any deterministic problem derived from those inputs will return after every restart.

Where the report starts making assumptions

The report becomes speculative around this conclusion:

"The failure appears to be entirely inside the FreeBSD kernel's carp/pfsync send path."

That statement is much stronger than the evidence supports.

The evidence shows:

No outbound pfsync packet was observed.

It does not yet show:

PF requested an eligible state update, pfsync accepted it, and the kernel failed while constructing or queuing the packet.

There is an important layer missing between those two conclusions.

The biggest missing question

The report never convincingly answers:

Should the primary have transmitted anything at all?

This is the central unresolved question.

pfsync is selective

pfsync does not blindly synchronize every state.

States can be excluded from synchronization. Relevant examples include:

• states created by rules using no-sync
• states imported from the peer
• certain internal or locally generated states
• states for which no synchronization update is required

Therefore:

"I created a state."

does not automatically mean:

"pfsync must send a state insert."

The report repeatedly treats those two statements as equivalent.

They are not.

The curl test is weak

The report states that new states were triggered using curl.

That test is only useful if the origin of the traffic is clearly specified.

If curl was executed on the firewall itself, the resulting state is locally generated traffic. That is not necessarily equivalent to ordinary forwarded client traffic passing through the firewall.

A much stronger test would be:

LAN client
    |
    v
Primary firewall
    |
    v
External destination

The operator should then inspect the resulting state:

pfctl -vvs states

and determine exactly which PF rule created it.

The test must establish all of the following:

1. A genuinely new state was created.
2. The state originated locally on the primary rather than being received through pfsync.
3. The state belongs to forwarded traffic through the firewall.
4. The exact rule that created it is known.
5. That rule does not use no-sync.
6. The state is otherwise eligible for pfsync transmission.

Without that chain, it has not been demonstrated that pfsync should have transmitted anything.

The missing PF rule inspection

This is arguably the largest omission in the report.

I expected to see:

pfctl -vvs rules

showing:

• the matching rule
• evaluation counters
• state creation counters
• whether no-sync is present

Instead, the report mostly states that the rules were checked in general.

That is insufficient.

The investigation needs to identify the exact rule responsible for the exact state that is expected to synchronize.

The report jumps over the PF layer

The reasoning effectively becomes:

No packets observed

↓

Not the network

↓

Not the driver

↓

Therefore kernel bug

But there is another major layer in between:

PF

↓

Should synchronization have been requested?

That layer has not yet been convincingly eliminated.

Bulk synchronization reasoning

Another place where the reasoning becomes shaky.

The report repeatedly says:

Every bulk transfer fails.

That is useful.

But it then concludes:

Therefore the master is unable to send.

That does not necessarily follow.

Bulk synchronization has two roles:

• requester
• responder

Those are different.

If the primary recreates pfsync0, then the primary becomes the requester.

In that situation, the backup is expected to send its state table.

That actually matches the observations.

To prove that the primary cannot answer a bulk request, the opposite direction should also be tested:

The backup should initiate the bulk request so that the primary must become the responder.

The report does not consistently distinguish these two cases.

The EVPN discussion

Large portions of the report discuss:

• VXLAN
• EVPN
• FRR
• Linux bridge forwarding
• bridge FDB learning
• RSS
• multiqueue virtio

Those are reasonable things to investigate.

However, they become largely irrelevant if this statement is correct:

Nothing is observed on pfsync0.

If nothing is visible on pfsync0, then those lower network layers cannot explain the absence of a packet that supposedly never left the pseudo-interface.

The reproduction is not actually a reproducer

The report describes:

Bring up two nodes.

Wait.

Observe the node that exhibits the bug.

That is not a reproducible procedure.

It describes one affected installation.

A useful reproducer would demonstrate how another developer can reliably trigger the problem on a clean installation.

The biggest logical leap

This sentence is speculative:

"Something prevents pfsync_out/the carp bulk-request logic from ever constructing or queuing an outbound frame."

Nothing presented actually demonstrates:

• pfsync_out()
• queueing failure
• mbuf allocation failure
• kernel transmit path failure

Only this has been demonstrated:

No outbound packets have been observed.

Those are very different conclusions.

What should be tested next

Before moving to DTrace or kernel instrumentation, I would first request:

1. Verify that the state is actually eligible for synchronization

pfctl -vvs states

Identify the exact state.

2. Identify the responsible PF rule

pfctl -vvs rules

Verify that:

• no-sync is absent
• the rule created the state
• the counters increase as expected

3. Use forwarded client traffic

Instead of running:

curl on the firewall

generate traffic from a LAN client through the firewall.

4. Reverse the bulk synchronization direction

Make the backup request the primary's state table.

Observe whether the primary responds.

5. Test a fresh OPNsense installation

Restore the same configuration into a newly installed VM.

Determine whether the behaviour follows:

• the configuration
• the VM
• the host
• or disappears entirely

6. Test on a plain Layer 2 network

Temporarily remove:

• VXLAN
• EVPN
• VRF
• overlay networking

to reduce variables.

Overall conclusion

The report contains excellent diagnostics, but the final conclusion is stronger than the evidence currently supports.

The observations convincingly support:

• Backup transmits pfsync traffic.
• Primary does not appear to transmit pfsync traffic.
• Bulk synchronization fails.
• The behaviour survives a full VM restart.

However, they do not yet prove that the FreeBSD kernel's transmit path is defective.

The report has not yet demonstrated that the primary generated a state that was actually eligible for synchronization, nor has it eliminated PF-level reasons why synchronization may never have been requested in the first place.

The next step should therefore not be kernel tracing.

The next step should be to correlate:

1. a newly created forwarded state,
2. the exact PF rule that created it,
3. confirmation that the rule is sync-eligible,
4. and only then verify whether pfsync transmission is attempted.

If that chain is demonstrated and the transmit path still remains completely silent, then the report becomes substantially stronger evidence of a genuine FreeBSD/OPNsense bug.

[MikeBishop]

UPDATE — root cause found and fix confirmed: OPNsense copies syncdev MTU onto pfsync0 uncapped, exceeding the mbuf allocator's ceiling

Found it, and confirmed the fix in production. This was never a transport, driver, EVPN, or CARP-role problem, and the earlier framing in this thread ("node's send path permanently disabled after a failed bulk transfer") had cause and effect backwards. The bulk failures were a symptom of a kernel allocator limit, triggered by an OPNsense config-generation bug. Correcting the record below.

Root cause, part 1: the kernel allocator ceiling (upstream issue)

pfsync_sendout() allocates its outgoing packet with:

m = m_get2(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR);
if (m == NULL) {
        if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1);
        V_pfsyncstats.pfsyncs_onomem++;
        return;
}

m_get2() (sys/kern/kern_mbuf.c) refuses anything above MJUMPAGESIZE, which is PAGE_SIZE = 4096 on amd64:

if (size <= MCLBYTES)
        return (uma_zalloc_arg(zone_pack, &args, how));
if (size > MJUMPAGESIZE)
        return (NULL);

pfsync_q_ins() accumulates b_len right up to pfsync0's configured MTU before flushing. With pfsync0 at 8930, every flush requests ~8946 bytes, gets NULL, and returns. No pfsync packet is ever constructed.

Root cause, part 2: why pfsync0 was at 8930 in the first place (actionable OPNsense bug)

pfsync0 doesn't inherit its MTU from the syncdev automatically — OPNsense copies it explicitly, every time the HA/interface config runs, in interfaces_pfsync_configure() (src/etc/inc/interfaces.inc):

/*
 * Fix presumed replication issue when outgoing MTU
 * is smaller than what is set in pfsync0, also see
 * https://github.com/opnsense/core/commit/15acbad935
 */
$intf_stats = legacy_interface_details($syncdev);
if (!empty($intf_stats['mtu'])) {
    mwexecf('/sbin/ifconfig pfsync0 mtu %s', [$intf_stats['mtu']]);
}

This copy is unconditional and uncapped. Any operator who puts a jumbo-framed interface on pfsync duty — a legitimate, unremarkable choice, done here for consistency with other interfaces on the same vswitch — gets pfsync0 pushed above MJUMPAGESIZE on every config reconfigure, permanently breaking pfsync transmission. The referenced commit (15acbad935) was fixing the opposite case (pfsync0 MTU too large relative to the outgoing path) and overshot into a range the allocator can't serve at all. This makes it squarely an OPNsense core issue, not only an upstream FreeBSD one.

Why the failure is permanent once triggered

pfsync_sendout()'s NULL branch returns without resetting b->b_len. The reset (b->b_len = PFSYNC_MINPKT) only happens after a packet is successfully built. The only other reset path is pfsync_drop(), reachable only from pfsync_clone_destroy() — and pfsync0 cannot be destroyed and recreated at runtime (ifconfig pfsync0 destroy → SIOCIFDESTROY: Invalid argument; it's a minimum-instance clone).

So once b_len exceeds ~4080 bytes it never recovers on its own: every insert pushes it higher, every flush fails identically, and neither a config reapply nor an in-place MTU change clears it — only recreating the interface (i.e., a reboot) resets b_len to a working state. This is what made the failure look permanent and role-independent throughout this thread, and why our earlier attempts to fix it live (ifconfig pfsync0 create ..., HA config reapply, a full VM restart without first correcting the MTU) all failed identically: none of them cleared b_len.

How this reconciles every observation from this thread

Observation	Explanation
pfsync_tx never called (DTrace, 0 hits across all windows)	pfsync_sendout bails at m_get2 before ever reaching a code path that calls tx
Nothing captured on the pfsync0 pseudo-interface itself	bpf_mtap() runs after m_get2(); a failed allocation is invisible even to a capture on pfsync0
pfsync_q_ins firing steadily while tx stayed at 0	Queuing was fine; only the flush was broken
Bulk transfer failed every attempt, ~65–90s, on both nodes	The bulk request itself could never be transmitted
Node sent fine while idle/BACKUP, died on becoming MASTER	Not role — traffic volume; idle kept b_len under the threshold
netstat -m showed 0 mbuf/cluster denials	m_get2 returns NULL on the size check before touching UMA, so no allocation failure is ever recorded there
9247565373 failures due to mbuf memory error	This is pfsyncs_onomem, and it was accurate — one increment per failed flush. Earlier in this thread it was dismissed as a bogus/wrapped counter because netstat -m was clean; that dismissal sent the investigation down a long detour before the two counters were correctly reconciled
A first attempt at lowering the physical interface's MTU didn't fix it	pfsync0's MTU only gets copied from the syncdev when interfaces_pfsync_configure() runs (HA reconfigure or boot) — changing the syncdev alone doesn't touch pfsync0 until then, and even after it does, the already-latched b_len needs a fresh clone instance (reboot) to clear

Fix, confirmed in production

Lower the syncdev's MTU below 4096 (comfortably under MJUMPAGESIZE - max_linkhdr, ~4080) and reboot each node so interfaces_pfsync_configure() re-copies the new, safe MTU onto a freshly cloned pfsync0 with b_len reset to PFSYNC_MINPKT:

ifconfig vtnet5 mtu 1500      # persist via the interface config, not just live
# reboot each node, one at a time

Post-fix, on primary, immediately after boot:

netstat -s -p pfsync
                0 failures due to mbuf memory error     ← was climbing continuously before
                11827 state inserts sent                ← climbing steadily, was frozen before
dmesg -a | grep -i "pfsync bulk"
[8]  carp: demoted by 240 to 240 (pfsync bulk start)
[10] carp: demoted by -240 to 0 (pfsync bulk done)      ← "bulk done", not "bulk fail", first attempt
[13] carp: demoted by 240 to 240 (pfsync bulk start)
[14] carp: demoted by -240 to 0 (pfsync bulk done)

state inserts sent climbs continuously under load, mbuf memory error stays at 0, and every bulk attempt at boot completes (bulk done) rather than timing out (bulk fail) — which also clears the CARP demotion the intended way and stops the full-cluster mastership handoff cycle documented earlier in this thread.

Suggested upstream fixes

1. OPNsense core (interfaces_pfsync_configure(), src/etc/inc/interfaces.inc): clamp the MTU value copied onto pfsync0 to a safe maximum (MJUMPAGESIZE - max_linkhdr, ~4080) instead of copying the syncdev's MTU verbatim. This is the actionable fix — it prevents the misconfiguration from being reachable via the GUI/config path at all, regardless of what the underlying FreeBSD allocator does.
2. FreeBSD (sys/netpfil/pf/if_pfsync.c): reject an MTU above the allocator's ceiling at SIOCSIFMTU time, so a manually-set oversized MTU fails loudly at config time instead of silently at runtime.
3. FreeBSD (same file): reset b_len (or drop the queue) on the m_get2 failure path, so a transient or persistent allocation failure degrades gracefully instead of latching the interface into a permanently non-transmitting state until next boot.

(1) is the one that actually caused this incident and is the most valuable to land; (2) and (3) are defense-in-depth against the same class of bug recurring via a different path.

Prior art

opnsense/core#4540 — "pfsync not working on a lagg if MTU is bigger than 1500" is the same underlying bug, reported against 20.7 and tagged upstream/third party issue. It appears never to have been root-caused, and the interfaces.inc MTU-copy behavior is still present unchanged in 26.1. This report supersedes it with a source-level cause, a mechanism for the "permanent until reboot" behavior, and a confirmed-in-production fix.

Environment

• OPNsense 26.1.11_10 (amd64), FreeBSD 14.3-RELEASE-p16, stable/26.1-n272152-9b6eef552f24
• Two Proxmox/KVM VMs, one per host, virtio NICs
• pfsync interface vtnet5, mtu was 8930, now 1500; dedicated point-to-point /30, pfsyncversion 1400, pfsyncdefer 0
• pfsync0: syncok: 1 throughout the entire incident — the flag was never implicated

---

Diagnosis was done collaboratively with AI assistance (Claude), across two models — Claude Sonnet for the interactive troubleshooting, Claude Opus for the final source-level analysis once the fault was narrowed to a single function. Several intermediate hypotheses were proposed and disproven along the way — packet-capture direction misreads, a virtio multiqueue/RSS theory, a prediction that a full VM restart would clear it, and a CARP-role-dependence theory — before the source-level cause above was identified, cross-checked against the actual kernel source for this build, and then confirmed by the person running the fix in production and reporting back the resulting counters and logs. All commands were run by hand on live hardware; all counters, DTrace output, and log excerpts quoted are real and were not edited or summarized from paraphrase.

[Monviech]

I would frame this as upstream, as the configuration should be rejected by if_pfsync if it's completely invalid.

Maybe try to open a report here:
https://bugs.freebsd.org/bugzilla/

Clamping anything on our end doesn't fix the issue, only the symptom. Documentation could help if somebody offers something here:
https://github.com/opnsense/docs

[MikeBishop]

That's fair, and I will file the issue upstream. Hopefully they fix it in a way that either removes the incompatibility with jumbo frames or silently caps itself at an MTU it can tolerate.

If they fix it in a way that noisily rejects invalid values, then I think there will still be an issue for OPNsense to address here as it's attempting to set one.

[Monviech]

Yeah if Jumbo Frames would be allowed in general that would be the best fix indeed.