When juggling multiple applications in Kubernetes, it's not uncommon to end up with all kinds of conflicting requirements. HTTP/HTTPS traffic is the easiest, since you can use something like Traefik (even if it does become more complicated if you run multiple endpoints), but if you want to run services that run other kinds of traffic.... It's actually a great reason to run MetalLB, as previously mentioned. The catch is, once the system start assigning different IPs to different services, how do you know which IP to contact? One option is to just use hard-coded IPs for everything, but that's not very scalable. Which is where you can have fun with something like ExternalDNS, which is able to register services with a DNS. In our case, using PowerDNS hosted on Kubernetes ends up being a very interesting option, allowing for everything to be internalized (although giving PowerDNS itself a static IP is a good idea!).
PowerDNS
Setting up PowerDNS isn't too bad if you already have a database set up (by default, I would recommend setting up an external database so that you don't need to worry about database corruption in case of a pod being forcibly stopped). The YAML file looks something like this (there is no official Helm chart as of this writing):
kind: Secret
metadata:
name: powerdns-secret
namespace: kube-system
type: Opaque
data:
PDNS_APIKEY: <base64 secret>
MYSQL_PASS: <base64 secret>
PDNSADMIN_SECRET: <base64 secret>
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: powerdns
namespace: kube-system
labels:
app: powerdns
spec:
replicas: 1
selector:
matchLabels:
app: powerdns
template:
metadata:
labels:
app: powerdns
spec:
containers:
- name: powerdns
image: pschiffe/pdns-mysql:alpine
livenessProbe:
exec:
command: ["/bin/sh", "-c", "pdnsutil list-zone <internal domain> 2>/dev/null"]
readinessProbe:
exec:
command: ["/bin/sh", "-c", "nc -vz <database hostname> 3306"]
initialDelaySeconds: 20
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "a=0;while [ $a -lt 200 ];do sleep 1;a=$[a+1];echo 'stage: '$a;if nc -vz <database hostname> 3306;then (! pdnsutil list-zone <internal domain> 2>/dev/null) && pdnsutil create-zone <internal domain>;echo 'End Stage';a=200;fi;done"]
env:
- name: PDNS_api_key
valueFrom:
secretKeyRef:
name: "powerdns-secret"
key: PDNS_APIKEY
- name: PDNS_master
value: "yes"
- name: PDNS_api
value: "yes"
- name: PDNS_webserver
value: "yes"
- name: PDNS_webserver_address
value: 0.0.0.0
- name: PDNS_webserver_allow_from
value: 0.0.0.0/0
- name: PDNS_webserver_password
valueFrom:
secretKeyRef:
name: "powerdns-secret"
key: PDNS_APIKEY
- name: PDNS_default_ttl
value: "1500"
- name: PDNS_soa_minimum_ttl
value: "1200"
- name: PDNS_default_soa_name
value: "ns1.<internal domain>"
- name: PDNS_default_soa_mail
value: "hostmaster.<internal domain>"
- name: MYSQL_ENV_MYSQL_HOST
value: <database hostname>
- name: MYSQL_ENV_MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: powerdns-secret
key: MYSQL_PASS
- name: MYSQL_ENV_MYSQL_DATABASE
value: powerdns
- name: MYSQL_ENV_MYSQL_USER
value: powerdns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 8081
name: api
protocol: TCP
- name: powerdnsadmin
image: aescanero/powerdns-admin:latest
livenessProbe:
exec:
command: ["/bin/sh", "-c", "nc -vz 127.0.0.1 9191 2>/dev/null"]
initialDelaySeconds: 80
readinessProbe:
exec:
command: ["/bin/sh", "-c", "nc -vz <database hostname> 3306 2>/dev/null "]
initialDelaySeconds: 40
env:
- name: PDNS_API_KEY
valueFrom:
secretKeyRef:
name: "powerdns-secret"
key: PDNS_APIKEY
- name: PDNSADMIN_SECRET_KEY
valueFrom:
secretKeyRef:
name: "powerdns-secret"
key: PDNSADMIN_SECRET
- name: PDNS_PROTO
value: http
- name: PDNS_HOST
value: 127.0.0.1
- name: PDNS_PORT
value: "8081"
- name: PDNSADMIN_SQLA_DB_HOST
value: <database hostname>
- name: PDNSADMIN_SQLA_DB_PASSWORD
valueFrom:
secretKeyRef:
name: powerdns-secret
key: MYSQL_PASS
- name: PDNSADMIN_SQLA_DB_NAME
value: powerdns
- name: PDNSADMIN_SQLA_DB_USER
value: powerdns
ports:
- containerPort: 9191
name: pdns-admin-http
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: powerdns-service-dns
namespace: kube-system
annotations:
metallb.universe.tf/address-pool: <IP identifier>
labels:
app: powerdns
spec:
type: LoadBalancer
ports:
- port: 53
nodePort: 30053
targetPort: dns
protocol: UDP
name: dns
selector:
app: powerdns
---
apiVersion: v1
kind: Service
metadata:
name: powerdns-service-api
namespace: kube-system
labels:
app: powerdns
spec:
type: ClusterIP
ports:
- port: 8081
targetPort: api
protocol: TCP
name: api
selector:
app: powerdns
---
apiVersion: v1
kind: Service
metadata:
name: powerdns-service-admin
namespace: kube-system
labels:
app: powerdns
spec:
type: ClusterIP
ports:
- port: 9191
targetPort: pdns-admin-http
protocol: TCP
name: pdns-admin-http
selector:
app: powerdns
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: powerdns
namespace: kube-system
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/frontend-entry-points: http,https
traefik.ingress.kubernetes.io/redirect-entry-point: https
labels:
network: internal
spec:
rules:
- host: powerdns.<internal domain>
http:
paths:
- path: /
backend:
serviceName: powerdns-service-admin
servicePort: 9191
Filling in all of the entries sets up a PowerDNS service backed by MySQL or MariaDB, along with the PowerDNS-Admin frontend.
ExternalDNS
After this is a matter of setting up ExternalDNS so that it talks to PowerDNS, for which there is a Helm chart:
kind: HelmChart
metadata:
name: external-dns
namespace: kube-system
spec:
chart: https://charts.bitnami.com/bitnami/external-dns-2.20.5.tgz
set:
provider: pdns
pdns.apiUrl: http://powerdns-service-api.kube-system.svc
pdns.apiPort: "8081"
pdns.apiKey: "<unencrypted PDNS_APIKEY from above>"
txtOwnerId: "external-dns"
domainFilters[0]: "<internal domain>"
interval: 10s
rbac.create: "true"
Once this is up and running, it will start registering services and ingresses with PowerDNS so that you can start querying the static IP specified earlier to find out IPs for various services, using their native ports (such as setting up an SSH server that will actually listen on port 22).
Next steps
After this is the obvious step: setting up DNS delegation for the specified subdomain. But that part should be easy, right? If you need to, take a look (again) at PowerDNS, except at the Recursor rather than the Authoritative Server.