Kubernetes services over BGP with Metallb and pfSense

Setting up a load balancer in a homelab is essential if you intend to expose your services internally or externally. While it’s worth noting that you don’t necessarily need a load balancer and could expose native Kubernetes services using tools like Cloudflare, that’s a topic for another blog post. However, having a load balancer becomes necessary if you’re tinkering with or learning Kubernetes APIs like Ingress or the Gateway API. This post will walk you through how I set up my load balancer.

Network Setup Overview

Here’s the context of my network setup:
I want to expose Kubernetes services through a load balancer using an IP range different from my home lab’s primary IP range. Additionally, I need these dynamically assigned IPs to be accessible from my home network. This configuration lets me use pfsense’s NAT capabilities to expose services to the outside world.

Versions of the software stack

Kubernetes - 1.31.x
pfSense - 2.7.2-RELEASE
frr (pfSense package) - frr9-9.0.2
metallb helm chart - 0.14.9

Below is a diagram illustrating the architecture:

Why Use BGP?

To achieve this setup, I chose to configure Border Gateway Protocol (BGP). BGP is a routing protocol commonly used to exchange routing information between networks. In this case, it enables seamless communication between the load balancer and the rest of my network. By leveraging BGP, I can dynamically advertise service IPs from my Kubernetes cluster to my home network.

Configuration Steps

Broadly speaking, setting up BGP involves two main steps:

Configuring pfSense:
The pfSense side of the setup is straightforward and entirely UI-driven. I’ll provide detailed instructions along with screenshots to guide you through the process.
Configuring Kubernetes:
On the Kubernetes cluster side, you’ll need to configure your load balancer solution. In this case, I’m using Metallb as the load balancer; see here for more details https://metallb.universe.tf/, which supports BGP. I’ll include a working example configuration that you can adapt for your environment.

Prerequisites

Before diving into the setup, make sure you have the following:

A working pfSense instance configured as your home network’s firewall/router.
A Kubernetes cluster running in your homelab.
Basic familiarity with networking concepts such as IP ranges, NAT, and routing protocols like BGP.
Administrative access to both pfSense and your Kubernetes cluster.

Configuring pfSense

Install the frr package.
- Go to System > Package Manager
- Select the Available Packages tab
- Search for the package frr
- Click + Install

Enable FRR
- Go to Services > FRR Global/Zebra
- Select Enable FRR
- Set Master Password to anything you like, it’s not needed in any other configs

Add ALLOW-ALL to RouteMap
- Go to Services > FRR Global/Zebra and to the RouteMap tab
- Select + Add
- Name ALLOW-ALL
- Description Match any route
- Action Permit
- Sequence 100
  Note: This change came with FRR 7.5, which requires explicit route acceptance

Enable BGP Routing
- Go to Services > FRR BGP and to the BGP tab
- Select Enable BGP Routing
- Set Local AS as 64500

Disable eBGP require policy
- Go to Services > FRR BGP and to the Advanced tab
- Select Disable the requirement to apply incoming and outgoing filter to eBGP sessions under eBGP section

Add neighbours, i.e. your Kubernetes nodes.
- Go to Services > FRR BCP and to the tab Neighbours
- Select + Add
- Fill in the General Options as below
  - Name Hostname or IP address
  - Description Hostname or IP Address
- Fill in the Basic Options
  - Remote As 64501
- Fill in the Peer Filtering
  - Route Map Filters with ALLOWED-ALL rule created above in step 3
    Repeat the step for each node. This could be optimised with setting up just once on the control plane, but I haven’t got around to it yet

These steps conclude the setup of the pfSense box for BGP routing. We will revisit to check the status and if everything is working as intended during the verification stage.

Configuring Kubernetes

Create a metallb-config.yaml. This file will contain three separate Kubernetes objects to configure your metallb to act as a BGP-capable load balancer.
- IP Address Pool: This object reserves a block of IP addresses that will be allocated when a Kubernetes service of type LoadBalancer is requested.
- BGP Peer Configuration: This object sets up the load balancer side of the BGP configuration, using values that correspond to your pfSense setup.
- BGP Advertisement Policy: This object controls how allocated addresses are advertised to peers, in this case, your pfSense router.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: bgp-address-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.10.101-192.168.10.250   # Loadbalancer range to be allocated
  autoAssign: true
---
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: bgp-peer
  namespace: metallb-system
spec:
  myASN: 64501                      # ASN for MetalLB
  peerASN: 64500                    # ASN of the pfSense BGP Routing
  peerAddress: 192.168.1.1          # IP address of the pfSense router
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgp-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
	- bgp-address-pool             # Advertise the address pool to BGP router 
  aggregationLength: 32            # Advertise individual IPs as /32 routes

Install metallb using the helm chart with the above created configuration file

helm repo add metallb https://metallb.github.io/metallb    
helm install metallb metallb/metallb -n metallb-system --create-namespace 
kubectl apply -f metallb-config.yaml

Verify BGP neighbours are talking

On Kubernetes
- Check if the daemon set deployed for metallb has started without any errors
- Make sure the configuration applied works without errors
On pfSense
- Go to Services > FRR BGP and to the Status tab
- Check the BGP Summary section. If the metallb neighbours (individual nodes) have connected to pfSense, you will see time connected in the Up/Down section, see below marked in green. If the neighbours have not been connected, you will see never status as marked in red.

Test & Verify

Create a sample nginx pod and expose it as a service of type LoadBalancer

kubectl run nginx --image nginx
kubectl expose pod nginx --name="nginx-lb" --selector="run=nginx" --port=80 --target-port=80 --type=LoadBalancer

Check if you see External-IP assigned to the nginx-lb service when you list the services

$ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1        <none>           443/TCP        680d
nginx        ClusterIP      10.98.128.241    <none>           80/TCP         2d21h
nginx-lb     LoadBalancer   10.109.249.119   192.168.10.102   80:32358/TCP   30h

On pfSense, go to Services > FRR BGP and to the Status tab. Check the BGP Routes section, and it should list your service as below.

Final setup: verify you can access the nginx site from another device on your home network.

February 11, 2025