Skip to content

refactor(cni): extract galactic-bgp as its own CNI chain plugin - #305

Open
privateip wants to merge 1 commit into
refactor/cni-chain-1-galactic-ipamfrom
refactor/cni-chain-2-galactic-bgp
Open

refactor(cni): extract galactic-bgp as its own CNI chain plugin#305
privateip wants to merge 1 commit into
refactor/cni-chain-1-galactic-ipamfrom
refactor/cni-chain-2-galactic-bgp

Conversation

@privateip

@privateip privateip commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Stack (merge bottom to top):


Summary

Third branch in the CNI plugin-chain split stack (based on #304). galactic-bgp is now its own chained CNI plugin, invoked last in the conflist after the master plugin and galactic-ipam, instead of in-process BGP/SRv6/eBPF publishing inside galactic-cni/galactic-tap-cni.

What moved

internal/cnibgp mirrors cniipam's shape: RunPlugin(), PluginConf{VPC, VPCAttachment, Namespace} parsed from stdin. It reuses internal/config.CNIConfig's GALACTIC_CNI_* env vars — shared node-level settings, not a BGP-specific flag.

galactic-bgp never touches the container netns. It learns interface kind (veth vs tap) and the allocated IPAM result from prevResult alone (prevresult.go): two interfaces means veth, one means tap. It publishes BGP state, then passes prevResult through unchanged as the last plugin in the chain.

cmdCheck is new: verifies the BGPVRFInstance/BGPAdvertisement CRDs and the eBPF vrf_table entry. cmdStatus probes the API server, matching STATUS across the chain. cmdDel stays a no-op — orphaned VRF/CRD state is reconciled by galactic-router's GC controller, not by any CNI DEL path.

Rollback scoping

resourceTracker in cnibgp/resource.go covers only what galactic-bgp's own ADD creates: BGPVRFInstance, BGPAdvertisement, the eBPF vrf_table entry. internal/cni/resource.go's tracker drops to vpc, vpcAttachment, vrfCreated, routesCreated and no longer needs a Kubernetes client.

hostgw extraction

ConfigureHostGateway configures the container's default route — kernel-interface work galactic-bgp shouldn't depend on. It now lives in internal/cni/hostgw, called directly by both master plugins before handing off down the chain.

PrevResult vs RawPrevResult

types.PluginConf.PrevResult has JSON tag "-" and is never populated; only RawPrevResult map[string]interface{} is. Pre-existing library quirk, not introduced here — ops_check.go already reads RawPrevResult, and inferFromPrevResult follows the same pattern. Out of scope to fix.

Verification

🤖 Generated with Claude Code

@mattdjenkinson

Copy link
Copy Markdown

Reviewed this one as well. A few issues here, some of which seem worth blocking on.

The rollback context in internal/cnibgp/ops_add.go:47 is created once at the top of cmdAdd with a 10s timeout, but publishBGPState's retryK8sOps can take up to about 30s (three attempts, each with its own fresh 10s context, plus backoff). If the k8s API is slow enough that retries run out before a hard failure surfaces, the deferred tracker.cleanup(rollbackCtx) ends up calling Delete with a context that's already expired. That fails with "context deadline exceeded" rather than NotFound, so client.IgnoreNotFound doesn't catch it, and the BGPVRFInstance/BGPAdvertisement CRDs that were just created leak instead of rolling back. Separate but related: rollbackCancel() only gets called in the error branch, so every successful ADD leaks that context's timer for up to 10 seconds.

CHECK doesn't verify everything ADD writes. The new checkEBPFEntry in internal/cnibgp/ops_check.go:96 only reads back registry.VRF.Get, never the locator or function tables, and skips the nodeID range validation that ADD treats as a hard error. If the locator/function tables get corrupted or go missing while the VRF table survives, or if nodeID drifts out of range after the fact, CHECK still reports the attachment healthy. That's a false positive on the SRv6 datapath.

internal/cnibgp/prevresult.go:44 introduces a new hard version constraint that didn't exist before. inferFromPrevResult goes through type100.NewResult, which only accepts a cniVersion of exactly 1.0.0 or 1.1.0. Previously BGP publish got the IPAM result as a plain in-process Go value with no version dependency at all. Now any conflist using something else, 0.4.0 for instance, which the master plugins would happily print, makes galactic-bgp's ADD fail for every attachment in the chain.

DEL doesn't respect the conflist's actual version either. internal/cnibgp/ops_del.go:26 never parses args.StdinData and always prints 1.0.0, unlike internal/cni/ops_del.go, which only falls back to that on a parse failure and otherwise uses pluginConf.CNIVersion. Since pluginConf is never parsed here, DEL also never logs vpc/vpcAttachment, which makes failures harder to trace back to a specific attachment.

There's also an e2e coverage gap with a comment that no longer matches reality. tests/e2e/e2e_test.go:214 says eBPF registration happens "inline from galactic-tap-cni's own cmdAdd," but this PR moved that into the separately chain-invoked galactic-bgp binary, and the test still only execs /galactic-tap-cni directly, never /galactic-bgp. The test keeps passing because it only checks the printed CNI result's interfaces and IPs, but it's quietly lost all coverage of BGP CRD creation and eBPF vrf_table registration on the ADD path.

Interface-kind inference is fragile too. internal/cnibgp/prevresult.go:53 decides veth versus tap purely from the interface count (one means tap, two means veth), and the assumption that galactic-bgp always runs immediately after the master plugin is only written down in comments, never enforced in code. #306 is about to add galactic-route into this same chain. If it, or anything added later, changes the interface count galactic-bgp sees, veth/tap gets silently misclassified and the wrong eBPF redirect kind gets programmed instead of the call failing loudly.

Smaller things: internal/cni/resource.go:29 (and identically internal/cnitap/resource.go:26) still registers bgpv1alpha1.AddToScheme even though this PR removes every BGP CRD read/write from that package, and the comment justifying it is circular. No test in the new internal/cnibgp package exercises resourceTracker.cleanup's actual wiring, only the standalone unregisterEBPFDatapath function is tested directly, and the old test that covered this end to end (including a rollback collision race) wasn't replaced. publishResult and resourceTracker declare the same five fields and cmdAdd copies them one by one instead of embedding one in the other, so a future field added to one but not the other stops being tracked with no compiler error to catch it. And egressKindForInterfaceType's empty-string branch is dead code left over from when interface type was optional; its only caller now never produces an empty string.

The rollback-context bug and the lost e2e coverage for BGP and eBPF registration seem worth blocking on. The version-constraint and DEL-version issues are at least worth a comment, even if they turn out to be non-issues given how conflists actually get authored in practice.

Step 2 of the CNI plugin-chain split (galactic/plan-cni-plugin-chain):
pulls BGP/SRv6/eBPF publishing out of the veth and tap master plugins
into its own chained CNI binary, galactic-bgp, invoked last in the
conflist after galactic-cni/galactic-tap-cni and galactic-ipam.

internal/cnibgp is the new plugin package:
- cnibgp.go: RunPlugin() entrypoint (skel.PluginMainFuncs, ADD/DEL/
  CHECK/STATUS/VERSION), mirroring the shape of cniipam/cnitap.
- types.go, config.go: PluginConf{VPC, VPCAttachment, Namespace},
  parsed from stdin. Deliberately reuses internal/config.CNIConfig
  (GALACTIC_CNI_* env vars) rather than inventing its own prefix like
  galactic-ipam did — these are shared node-level settings (API server
  address, namespace, etc.), not a BGP-specific concern, so there is
  no reason to duplicate or rename them for this binary.
- prevresult.go: inferFromPrevResult() reconstructs everything
  galactic-bgp needs (interface kind — veth vs tap, and the allocated
  IPAM result) purely from the previous plugin's prevResult, since
  galactic-bgp itself never touches the container's network namespace.
  Interface kind is inferred from prevResult shape: two interfaces
  means veth (host+container), one means tap.
- resource.go: a resourceTracker scoped to exactly what galactic-bgp's
  own ADD creates — BGPVRFInstance, BGPAdvertisement, and the eBPF
  vrf_table entry. This is intentionally smaller than the old single
  process-wide tracker: the kernel-interface/VRF cleanup that used to
  live alongside these now belongs to each master plugin's own
  tracker (internal/cni, internal/cnitap), scoped to exactly what its
  own ADD creates. Selective rollback stays correct because each
  plugin only ever rolls back what it itself created.
- ops_add.go/ops_del.go/ops_check.go: cmdAdd parses config, infers
  from prevResult, publishes BGP state, and passes prevResult through
  unchanged (galactic-bgp is the last plugin in the chain). cmdDel is
  a no-op everywhere in the chain, as decided in the plan — orphaned
  VRF/CRD state is reconciled independently by galactic-router's GC
  controller, not by any CNI DEL path, so cmdDel does not need to
  distinguish "resources this plugin created" from anything else.
  cmdCheck is new logic (the old bgp.go had no CHECK story of its own):
  it verifies the BGPVRFInstance and BGPAdvertisement CRDs exist and
  cross-checks the eBPF vrf_table entry via a new checkEBPFEntry
  helper. cmdStatus probes the API server, matching the STATUS story
  every other binary in the chain now implements (per plan decision).

internal/cnibgp/bgp.go keeps the actual BGP/SRv6/eBPF logic moved over
from internal/cni/bgp.go, with everything now unexported since it is
package-internal to cnibgp rather than shared across internal/cni:
publishConfig, publishResult, publishBGPState, egressKindForInterface-
Type, unregisterEBPFDatapath, plus the untouched allocation/collision/
CRD-building helpers (allocateArgument, checkArgumentCollision,
lookupBGPRouter, buildVRFInstanceSpec, buildAdvertisementSpec,
ipamAdvertisementPrefixes, allAdvertisedPrefixes, registerEBPFDatapath,
isTransientError/retryK8sOps).

Design refinement beyond the original plan: ConfigureHostGateway and
its helpers (installGatewayNeighbor, ipv4GatewayAddrParams,
installGatewayRoute, routeConflicts) do NOT move into galactic-bgp.
They configure the container's default route to the VRF gateway
address, which is kernel-interface/netns work — exactly the kind of
dependency the plan's own rationale for splitting BGP out says
galactic-bgp should have zero of. Moving them into galactic-bgp would
have reintroduced that dependency one step later in the chain instead
of removing it. They now live in a new internal/cni/hostgw package,
called directly by both master plugins (galactic-cni and
galactic-tap-cni) right after they configure the container interface,
before handing off down the chain.

Also discovered while wiring inferFromPrevResult: types.PluginConf's
PrevResult field (from containernetworking/cni/pkg/types) has a json
tag of "-" and is never populated by json.Unmarshal; only the sibling
RawPrevResult map[string]interface{} field (tag "prevResult,omitempty")
actually receives the previous plugin's result. This is a pre-existing
quirk of that library, not something introduced by this split, and
existing code elsewhere in this repo already works around it by
reading RawPrevResult directly (e.g. ops_check.go). inferFromPrevResult
follows that same existing pattern. Left as-is rather than fixed here,
since fixing it is unrelated to this split's scope.

Taskfile.yaml, containers/galactic-cni/Dockerfile, and
internal/installer/installer.go (SourceBGPBinary) gain galactic-bgp
alongside galactic-cni/galactic-tap-cni/galactic-ipam, following the
exact same pattern established for those two in prior steps.

internal/cni/resource.go's resourceTracker drops all BGP/eBPF fields,
leaving only vpc, vpcAttachment, vrfCreated, routesCreated — cleanup()
no longer needs a Kubernetes client at all, since it never touches BGP
CRDs. internal/cni/result.go's buildVethResult calls hostgw.Configure-
HostGateway directly and returns only an error rather than threading
an IPAMResult/MAC address back up for the caller to hand to a bgp
helper that no longer lives in this package. Mirrored identically in
internal/cnitap.

Verification: task lint (0 issues), task build, task test:unit all
green. task test:e2e not run in this step (matches prior two steps in
this stack — requires sudo modprobe vrf plus a Kind cluster bring-up,
deferred to the end of the full stack per the plan's verification
approach).
@privateip
privateip force-pushed the refactor/cni-chain-2-galactic-bgp branch from e33092c to 956b547 Compare August 8, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants