Makeshift Kubernetes external load balancer with haproxy

Some time ago I've replaced Google Analytics with Plausible. It works great, except for one tiny thing. The map of visitors was empty. Due to various layers of Network Adress Translations in k3s networking setup, the original client IP address information was not reaching analytics engine.

There are solutions – there is a PROXY Protocol exactly for that case. And Traefik, which handles ingress in k3s, supports PROXY. Only a bit of gymnastic was needed.

Legacy IPv4 traffic entry point to my bare-metal cluster has a form of a small in-the-cloud virtual machine. It routes incoming TCP/443 traffic over the VPN into the cluster. The VM itself is not a part of kubernetes setup – I cannot run any pods on it. I've decided to use Ansible to configure it.

The outcome lives in k8s-haproxy-external-lb and gives me following map:

(greetings Australia, have you found information about the red LED on Sonoff?)

There are few moving parts, but with Python, Kubernetes and Ansible, the result is suprisingly simple:

• there's a persistent pod running on the k8s cluster, watching EndPoints exposed by Traefik. When a change occurs (traefik pod restart, replica count modification, etc.) – ansible playbook is triggered. This pod may be seen as a k8s controller.

• ansible playbook collects Traefik pod's IP addresses and ports. JSON parsing in ansible is a bit suboptimal: … | first | first | first looks bad but works.

• still using ansible, haproxy configuration file is created, put on the edge nodes, and the service is restarted. I've selected haproxy because:

  • it implements PROXY protocol

  • I did similar thing on OpenShift

• haproxy passes received traffic directly to Traefik. This happens at the TCP level. TLS is terminated at Traefik, and certificates do not leave kubernetes cluster.

Some minimal preparations were needed. Communication between edge node and kubernetes pod network had to be established. This was done in an instant, thanks to Wireguard. SSH keypair for ansible had to be put in a k8s Secret and distributed among edge nodes. Finally, small fix was needed in ansible itself: local connection plugin was not happy when run in a container, as random user without an entry in /etc/passwd.

Traefik had to be configured to trust PROXY protocol information and generate X-Forwarded-For headers. Plausible utilized information in those headers without additional tinkering.

Configuration details are described at https://github.com/zdzichu/k8s-haproxy-external-lb.