A Highly Available Egress Gateway for Kubernetes with Cilium and kube-vip

If you've ever tried to whitelist a Kubernetes workload on a partner's firewall, you've probably run into this question: which IP address does my traffic actually leave the cluster with? The answer is rarely the one people expect, and fixing it properly takes two tools that, on paper, have nothing to do with each other.

One clarification up front, since it's in the title: "highly available" here means the classic active-passive kind, the same sense in which a VRRP/keepalived setup or kube-vip's own control-plane use case is highly available — exactly one gateway node active at a time, with automatic, unattended failover to another when it disappears. It does not mean zero-downtime. Failover takes a few seconds, and already-open connections through the old gateway don't survive it. If that distinction matters for your use case, the limitations section at the end spells out exactly what to expect during a handover.

Where does your traffic actually come from?

A pod has its own IP address. Inside the cluster, that's the address everything talks to — Services, other pods, network policies, all of it operate on pod IPs. So it's a natural assumption that this is also the IP an external server sees when your pod calls out to the internet.

It isn't. As soon as traffic leaves the cluster, most CNIs — Cilium included — masquerade it: the source address is rewritten (SNAT'd) to the IP of the node the pod happens to be running on. This is not a Cilium quirk, it's the same behavior kube-proxy's iptables rules have always implemented, just reimplemented in eBPF. External routers only know how to get packets back to your nodes, not into an internal pod CIDR that's meaningless outside the cluster, so the node IP has to become the source.

Two consequences follow from that:

• The IP an external system sees depends on which node the pod was scheduled to — not something you control, and not something that stays constant across a reschedule, a node drain, or a cluster autoscaling event.
• If your nodes only have private IPs (the common case in most on-prem and cloud setups), the traffic gets NAT'd again at your egress router or cloud NAT gateway, at which point every workload in the cluster starts sharing the same public IP with no way to tell them apart.

Why you'd want a static, public egress IP

None of this matters until you need to hand an IP address to somebody else's firewall. A partner API that only accepts requests from allow-listed IPs, a legacy system behind a strict perimeter, a compliance requirement that outbound traffic from a specific workload is traceable to one address — all of these want one thing: a single, stable, public IP that identifies this workload's traffic, independent of which node happens to be running it right now.

That's exactly the gap Cilium's Egress Gateway feature is built to close.

Cilium's Egress Gateway: the routing half of the problem

I've written before about consolidating a cluster onto Cilium, and Egress Gateway is one more reason I keep reaching for it instead of bolting on yet another single-purpose component. It's part of Cilium itself — no separate controller to run — and it does exactly what we need: pick pods by label, pick a destination CIDR, pick a gateway node and an egress IP, and Cilium's datapath takes care of routing and SNAT'ing matching traffic through that node with that IP, no matter where the pod itself is actually scheduled.

A CiliumEgressGatewayPolicy looks roughly like this:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-gateway-sample
spec:
  selectors:
    - podSelector:
        matchLabels:
          egress-gateway: "true"
  destinationCIDRs:
    - 0.0.0.0/0
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-gateway: "true"
    egressIP: "203.0.113.10"

Straightforward enough. The catch is one sentence, buried in the docs on selecting and configuring the gateway node:

The egress IP must be assigned to a network device on the node.

Cilium will not create, assign, or move that IP for you. It only ever uses an IP that's already sitting on an interface of the selected node. If the address isn't there, egress traffic for that policy is silently dropped. That single requirement drags in two problems we now have to solve ourselves:

1. Somebody has to put the IP on a node's interface — and take it off again and put it somewhere else when that node disappears (a reboot, a drain, a failed health check).
2. Somebody has to make that IP reachable from the internet, which — since we're not on the same L2 segment as our upstream router — means speaking BGP.

Cilium has no intention of building this itself. A feature request asking for exactly this — letting a CiliumEgressGatewayPolicy push its egressIP into BGP without the IP being locally assigned — was closed as not planned. Digging further, there's an older, still-open issue with essentially the same ask (Cilium should manage egress IP failover itself), where a maintainer confirms the community-contribution door is open but the Cilium team isn't building it — and, tellingly, someone links to Isovalent's commercial "Egress Gateway High Availability" feature, which is precisely this capability, sold as an enterprise add-on. Interestingly, in that same thread other users independently arrive at the exact workaround this article describes: bind a floating IP to a node with kube-vip and sync a node label to match it.

So: no native failover, no native BGP for the egress IP, and no plan to change that in open-source Cilium. Which means we need a second tool that's very good at exactly one job — owning a floating IP and telling the network where it currently lives.

kube-vip: the IP-management half

kube-vip does one thing: it makes a virtual IP highly available across a set of nodes, using Kubernetes leader election to decide which node currently owns it, and either ARP or BGP to make the rest of the network aware of that fact. It supports two independent modes:

• Control plane load balancing (cp_enable) — the classic use case: give the Kubernetes API server a floating IP across control-plane nodes, so losing one node doesn't take down kubectl or kubelet-to-API connectivity.
• Service load balancing (lb_enable) — watches Service objects of type LoadBalancer and assigns/announces IPs for them, the same role MetalLB plays (and, as it happens, the role I already replaced with Cilium's own LB-IPAM and BGP Control Plane in that earlier consolidation).

Neither mode is "manage an arbitrary floating IP for whatever I point you at" — but the control-plane mode is close enough. It doesn't actually care that the IP it's managing belongs to an API server; from kube-vip's point of view it's just: run leader election among a set of pods, bind an address to an interface on whichever one wins, and announce it. We're going to repurpose that machinery for our egress IP instead, with lb_enable: "false" since we have no interest in its Service-LB behavior here.

Closing the loop: getting Cilium and kube-vip to agree

kube-vip solves "which node has the IP right now." Cilium's nodeSelector needs to know which node that is. The missing piece is a live label on the winning node that Cilium can select on — and until fairly recently, kube-vip could only maintain that kind of label in its Service-LB mode, not in control-plane mode. That gap was closed by kube-vip PR #1566, which decouples node labeling from the Service object model and makes it available via a single flag, enable_node_labeling, regardless of which mode is driving the election. The fix shipped in kube-vip v1.2.0 — anything older will silently ignore enable_node_labeling under cp_enable, so pin at least that version. Whichever node currently holds the VIP gets labeled kube-vip.io/has-ip=<the-ip>; the label is removed the moment it loses the lease.

With that, all three pieces click together:

Cilium picks the gateway node by matching egress-gateway: "true" and kube-vip.io/has-ip: "203.0.113.10". In the common case, kube-vip's leader election keeps exactly one node carrying that label, so there's no ambiguity about which candidate node Cilium routes through. During an actual failover, though, that's only mostly true — see below for what actually happens in the handoff window.

Implementation

1. Cilium Helm values

Assuming a cluster already running Cilium as the kube-proxy replacement (see my earlier article if it isn't), enabling Egress Gateway only needs three values:

kubeProxyReplacement: true

bpf:
  masquerade: true

egressGateway:
  enabled: true

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set bpf.masquerade=true \
  --set egressGateway.enabled=true \
  --set kubeProxyReplacement=true

kubectl rollout restart daemonset cilium -n kube-system
kubectl rollout restart deployment cilium-operator -n kube-system

One prerequisite worth calling out explicitly: Egress Gateway requires identityAllocationMode: crd (the default in current Cilium) and is incompatible with kvstore-based identity allocation and with Cluster Mesh. If you're running an older config with a kvstore, that's a separate migration before any of this works.

2. The egress gateway policy

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-gateway-sample
spec:
  selectors:
    - podSelector:
        matchLabels:
          egress-gateway: "true"
  destinationCIDRs:
    - 0.0.0.0/0
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-gateway: "true"
        kube-vip.io/has-ip: "203.0.113.10"
    egressIP: "203.0.113.10"

One detail that's easy to get wrong, and that I got wrong myself on a first pass by reading kube-vip's source instead of running it: whether kube-vip.io/has-ip actually carries the IP, or comes out empty, depends on which environment variable configured the address. kube-vip has two separately-parsed fields that look almost identical — vip_address populates the field used for interface binding and BGP, while the node-labeling code reads a different field, populated only by a second, plainly-named variable: address. Set only vip_address (the obvious choice, and the one in kube-vip's own examples), and the VIP binds and announces exactly as expected — but the label ends up as kube-vip.io/has-ip= with no value, because the field the labeler actually reads was never touched. A matchLabels entry of kube-vip.io/has-ip: "" does still match that — Kubernetes label selectors are fine with an explicitly empty value — so a single-egress-gateway setup works either way. It falls apart the moment you want a second one: two different VIPs would both produce the same empty label, and Cilium would have no way to tell which node belongs to which policy. The robust fix is to also set address to the same value as vip_address, so the label carries the IP it actually represents:

3. The kube-vip DaemonSet

This is where the actual VIP management, BGP announcement, and node labeling happen:

apiVersion: v1
kind: Namespace
metadata:
  name: egress-gateway
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-vip-egress-gateway
  namespace: egress-gateway
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-vip-egress-gateway
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-vip-egress-gateway
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-vip-egress-gateway
subjects:
  - kind: ServiceAccount
    name: kube-vip-egress-gateway
    namespace: egress-gateway
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-vip-egress-gateway
  namespace: egress-gateway
spec:
  selector:
    matchLabels:
      app: kube-vip-egress-gateway
  template:
    metadata:
      labels:
        app: kube-vip-egress-gateway
    spec:
      serviceAccountName: kube-vip-egress-gateway
      hostNetwork: true
      nodeSelector:
        egress-gateway: "true"
      containers:
        - name: kube-vip
          image: ghcr.io/kube-vip/kube-vip:v1.2.1
          args: ["manager"]
          env:
            - name: cp_enable
              value: "true"
            - name: cp_namespace
              value: egress-gateway
            - name: lb_enable
              value: "false"
            - name: vip_interface
              value: "eth0"
            - name: vip_address
              value: "203.0.113.10"
            - name: address
              value: "203.0.113.10"
            - name: vip_subnet
              value: "32"
            - name: vip_arp
              value: "false"
            - name: vip_leaderelection
              value: "true"
            - name: vip_leasename
              value: kube-vip
            - name: vip_namespace
              value: egress-gateway
            - name: vip_nodename
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: bgp_enable
              value: "true"
            - name: bgp_routerinterface
              value: eth0
            - name: bgp_as
              value: "65010"
            - name: bgp_peers
              value: "203.0.113.1:65001::false,203.0.113.2:65001::false"
            - name: enable_node_labeling
              value: "true"
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
                - NET_RAW
                - SYS_TIME

A few of these deserve a comment:

• cp_enable: "true" / lb_enable: "false" — we're using kube-vip's control-plane election engine purely as a generic "elect one node, bind an IP" mechanism, not for its intended API-server purpose. Because of that, the ClusterRole above only grants nodes and leases — no services/endpoints. Those only get touched by kube-vip's Service-LB reconciliation loop, which never even starts while lb_enable is false; granting them here would just be unused blast radius.
• vip_arp: "false" combined with bgp_enable: "true" — we're not on the same L2 segment as our router, so BGP (not gratuitous ARP) is how the rest of the network learns where 203.0.113.10 currently lives.
• bgp_peers follows the format peer-ip:peer-asn:password:multihop — here two upstream routers, no password, single-hop.
• enable_node_labeling: "true" is the flag from PR #1566 — without it, none of the rest matters, because Cilium would have no way to find the node kube-vip just configured.
• address, set to the same value as vip_address — easy to skip since it looks redundant, but it's the only thing that gives the kube-vip.io/has-ip label an actual value instead of an empty one (see the callout above).
• Only nodes labeled egress-gateway: "true" run this DaemonSet at all — that's your pool of candidate gateway nodes:

kubectl label node worker-1 worker-2 worker-3 egress-gateway=true

4. Verifying it

A minimal test workload, scheduled everywhere via a DaemonSet so it doesn't matter which node actually runs it:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: egress-test
  namespace: egress-gateway
  labels:
    egress-gateway: "true"
spec:
  selector:
    matchLabels:
      egress-gateway: "true"
  template:
    metadata:
      labels:
        egress-gateway: "true"
    spec:
      containers:
        - name: curl
          image: curlimages/curl
          command: ["sleep", "3600"]

kubectl exec -n egress-gateway -it daemonset/egress-test -- curl -s ifconfig.io
203.0.113.10

No matter which node the test pod actually landed on, its outbound traffic leaves with the egress IP — and if you cordon and drain the current gateway node, kube-vip re-elects, re-labels, re-announces via BGP, and the same command keeps returning the same IP.

Where this approach still has sharp edges

This setup gets you a working, dynamically failing-over, BGP-announced egress IP without paying for an enterprise license — but it's two independently reconciling controllers wired together through a label, not one integrated feature, and that shows up in a few places:

• Existing connections don't survive a gateway change. When the winning node changes, Cilium's egress gateway BPF map entry for the policy changes with it — but already-established connections have conntrack/SNAT state tied to the old gateway, and that state isn't migrated. This is a known, still-discussed limitation in Cilium itself, not something specific to the kube-vip approach: any egress-gateway failover, however it's triggered, blackholes in-flight connections. New connections pick up the new gateway immediately; long-lived ones (an open TCP session to a partner API, say) need to reconnect.
• Failover is automatic, but neither instant nor atomic. Cilium itself famously does not fail over between multiple matching gateway nodes on its own — issue #18230 documents outages of several minutes when relying on Cilium alone to notice a dead gateway node. kube-vip's leader election fixes that, but binding the IP, withdrawing/announcing the BGP route, and patching the node label are three separate operations on kube-vip's own timeline, not one transaction. In testing, both the outgoing and incoming node ended up carrying kube-vip.io/has-ip=<ip> at the same time for roughly 3–4 seconds during a failover. Cilium doesn't treat that overlap as an error: per the docs, when a nodeSelector matches multiple nodes it deterministically picks the first one in lexical order by node name — so traffic can keep landing on the old node until its label is actually removed, not the moment the new node's label appears. The overall failover latency is bounded entirely by kube-vip's lease-duration/renew-deadline settings; Cilium has no say in how fast any of this happens, it only reacts to whatever labels exist at a given moment.
• One IP, one policy, one kube-vip VIP config. If you need several distinct public egress identities (e.g., one per tenant or per downstream partner), you're running multiple CiliumEgressGatewayPolicy and kube-vip VIP configurations side by side, not one shared setup.
• This is a comparatively young combination. enable_node_labeling is a recent addition to kube-vip; pin your versions, and actually test a node failure — don't just trust that the label converges the way the docs suggest.

If you'd rather not own this integration yourselves, it's worth knowing the enterprise version of Cilium (via Isovalent) sells exactly this as a packaged "Egress Gateway High Availability" feature. For everyone else, kube-vip plus a bit of glue is, as of today, the pragmatic way to get there in open-source Cilium.

If you haven't already, my previous article on consolidating a cluster onto Cilium — replacing kube-proxy, MetalLB, and ingress-nginx — covers the groundwork this setup builds on.