As previously mentioned, I've been using k3s in order to run an internal Kubernetes cluster. One thing that it doesn't do out of the box, however, is handle more than one IP, which can be restricting to Ingresses. In my case, I'm interested in providing services to a few different methods, so this would be problematic. This is something that can be addressed by using different software, however - in this case, a software implementation of a loadbalancer by the name of MetalLB.
Due to setting up MetalLB instead of k3s' default servicelb, this is an opportunity to tweak Traefik as well. This provides particularly useful due to wanting an instance of Traefik for each region. As such, the installation command turns into this (the wget version follows):
MetalLB is a fairly straightforward installation:
At this point, creating a simple configuration file enables MetalLB:
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: external
protocol: layer2
addresses:
- 1.2.3.4/32 # external IP
- name: internal
protocol: layer2
addresses:
- 192.168.0.1/32 # internal IP
auto-assign: false
This example just takes a single external and a single internal IP, naming them external and internal respectively (very imaginative, I know). The interesting point is the auto-assign, as it declares that this IP will not be automatically used. IP ranges can also be used if desired.
After this, Traefik (external) is fairly straightforward to set up as well, using a modified version of the YAML file bundled with k3s. We add a couple of bonuses while we're at it (full documentation available here):
kind: HelmChart
metadata:
name: traefik
namespace: kube-system
spec:
chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
set:
rbac.enabled: "true"
ssl.enabled: "true"
ssl.enforced: "true"
ssl.permanentRedirect: "true"
metrics.prometheus.enabled: "true"
kubernetes.labelSelector: network!=internal
kubernetes.ingressEndpoint.useDefaultPublishedService: "true"
acme.enabled: "true"
acme.staging: "false"
acme.email: <your e-mail address>
acme.challengeType: tls-alpn-01
acme.delayBeforeCheck: 90
acme.domains.enabled: "true"
The interesting points here are the kubernetes.labelSelector, as this declares that it should use non-internal addresses (in this case, 1.2.3.4), as well as enabling ACME for websites served from here. The ssl.* settings just build upon that.
The Traefik (internal) YAML looks fairly similar, although simplified due to not having any of the ACME settings:
kind: HelmChart
metadata:
name: traefik-internal
namespace: kube-system
spec:
chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
set:
fullnameOverride: traefik-internal
rbac.enabled: "true"
ssl.enabled: "true"
metrics.prometheus.enabled: "true"
kubernetes.labelSelector: network=internal
kubernetes.ingressEndpoint.useDefaultPublishedService: "true"
service.annotations.metallb\.universe\.tf/address-pool: internal
The name is different here, naturally, but there's also the fullnameOverride setting, used so that Kubernetes components don't collide with the "regular" Traefik. The kubernetes.labelSelector is different here, as you can see, and we take advantage of MetalLB's ability to request specific IPs in order to have Traefik's loadbalancer service use the internal IP. The backslashes allow for specifying raw periods in the annotation name.
At this point, the previous docker-demo.yaml's Service can be tweaked to the following:
.
.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
labels:
network: internal
spec:
rules:
- host:
http:
paths:
- path: /
backend:
serviceName: web
servicePort: 8081
For purpose of testing, we leave the host: entry blank so that it accepts all connections (yes, this could have been done with the previous example as well). The addition of the network: internal label means that this is exposed on 192.168.0.1 instead of 1.2.3.4. And that's it!