(Re-)Building the world

One of the things you really need with systems is backups. I've (unfortunately) been pretty bad about those too, so have taken advantage of the recent insanity to finally put together a backup routine. In this instance, the goal is to have one an onsite backup (effectively a snapshot) with offsite backups. It's a layered system, however, so requires a little bit of explanation. Note that this takes advantage of Kubernetes and the previously mentioned ExternalDNS setup for easily addressable hostnames.

Local

We want secure backups, so in this case, we're opting for gocryptfs in reverse mode in order to have straightforward secure backups, where the goals include:

Encrypted backups where it's okay for the local system has the key laying about (since it's the system that generates the data, after all), but the central store lacks that same key.
Reasonable space usage (i.e. doubling space requirements for encryption isn't fun).
Reasonable package overhead (e.g. having to run an entire Java engine on a lightweight system isn't fun).

We need gocryptfs installed from edge/testing:

# Enable the edge/testing repository
$ vi /etc/apk/repositories
$ apk add gocryptfs
# Disable the edge/testing repository if you're not living on it
$ vi /etc/apk/repositories

Then, set up the basics for the mount. For this, we'll be using /root/backup as our target directory:

$ mkdir /root/backup
$ gocryptfs -init -reverse backup
# Enter password here, record the master key.
$ mkdir /root/.gocryptfs
$ chmod go-rwx /root/.gocryptfs
$ mv /root/backup/.gocryptfs.reverse.conf /root/.gocryptfs/backup.conf

At this point, it's now possible to use gocryptfs' reverse mode in order to create an encrypted folder that uses no additional space:

$ mkdir /root/backup-crypt
$ modprobe fuse
$ gocryptfs -config /root/.gocryptfs/backup.conf -masterkey <master key> -reverse /root/backup /root/backup-crypt

This directory can be easily copied to another location.</p><p>But what about backup restores, you ask? The backup configuration will no longer exist. Conveniently, it's easy enough to regenerate the configuration (although you would be using the -reverse flag to generate the config). Just make sure you record the master key in a safe place.

Remote

The next step is creating a central location to store the files, where the goals include:

Encryption in transit (mostly to protect credentials, as the backups themselves are already encrypted).
Account isolation (i.e. one system is unable to access another system's backups).
Minimal leakage of credentials.

In this case, SFTP (via SSH) makes the most sense, particularly once ChrootDirectory and ForceCommand internal-sftp are enabled. On the client side, sftp -b allows for basic scripting. In this instance, this SFTP Docker container within Kubernetes works out well. The first step involves setting up local storage:

backup-rw.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: backup-rw-pv
labels:
   name: backup-rw-pv
spec:
capacity:
   storage: <storage>
volumeMode: Filesystem
accessModes:
  - ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
   path: <path>
nodeAffinity:
   required:
     nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
         operator: In
         values:
          - <system>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-rw-pvc
spec:
accessModes:
    - ReadWriteOnce
volumeMode: Filesystem
resources:
   requests:
     storage: <storage>
storageClassName: local-storage
selector:
   matchLabels:
     name: "backup-rw-pv"

Then, followed up by the actual application container:

backup-sftp.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: backup-sftp-users
data:
users.conf: |
   <user entries>
---
apiVersion: v1
kind: ConfigMap
metadata:
name: backup-sftp-init
data:
init-sftp.sh: |
   #!/bin/sh

   cat << EOF > /etc/ssh/ssh_host_ed25519_key
   -----BEGIN OPENSSH PRIVATE KEY-----
   <private key>
   -----END OPENSSH PRIVATE KEY-----
   EOF

   cat << EOF > /etc/ssh/ssh_host_ed25519_key.pub
   <public key>
   EOF

   cat << EOF > /etc/ssh/ssh_host_rsa_key
   -----BEGIN RSA PRIVATE KEY-----
   <private key>
   -----END RSA PRIVATE KEY-----
   EOF

   cat << EOF > /etc/ssh/ssh_host_rsa_key.pub
   <public key>
   EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-sftp
labels:
   app: backup-sftp
spec:
replicas: 1
selector:
   matchLabels:
     app: backup-sftp
template:
   metadata:
     labels:
       app: backup-sftp
   spec:
     containers:
      - name: backup-sftp
       image: atmoz/sftp:alpine
       volumeMounts:
        - name: sftp-users
         mountPath: /etc/sftp
        - name: sftp-init
         mountPath: /etc/sftp.d
        - name: backup-rw-pvc
         mountPath: /home
       ports:
        - containerPort: 22
     volumes:
      - name: sftp-users
       configMap:
         name: backup-sftp-users
      - name: sftp-init
       configMap:
         name: backup-sftp-init
         defaultMode: 0744
      - name: backup-rw-pvc
       persistentVolumeClaim:
         claimName: backup-rw-pvc
---
apiVersion: v1
kind: Service
metadata:
name: backup-sftp
annotations:
   external-dns.alpha.kubernetes.io/hostname: <hostname>
   metallb.universe.tf/address-pool: <pool>
labels:
   app: backup-sftp
spec:
type: LoadBalancer
ports:
    - port: 22
     targetPort: 22
selector:
   app: backup-sftp

This hooks up everything up neatly. The user entries follow this format:

user:pass[:e][:uid[:gid[:dir1[,dir2]...]]]

Best practices involve leaving pass empty and using SSH keys instead. Due to having to juggle permissions appropriately, SSH keys under the <path>/<user>/.ssh/keys directory are added to the authorized_keys file, so public keys should be added there. In order to have the container recognize new users, however, the container needs to be restarted:

Add a user entry to the YAML configuration.
Add the SSH public key to the correct directory (setting new directory permissions appropriately).
Ensure that Kubernetes has read the new YAML configuration.
Restart the SFTP pod (likely by killing the current pod).

Having client systems uploading their individual backups is done via a (simple?) script, probably located in somewhere like /etc/periodic/daily (so that it's automatically run nightly):

/etc/periodic/daily/backup

#!/bin/sh

TARGET_DIR=/var/backup

# Make local backups.
rm -rf ${TARGET_DIR}/*
<commands to generate the backup here>

GOCRYPT_TARGET_DIR=${TARGET_DIR}-crypt

# Make sure that the gocryptfs directory isn't still mounted from before (state reset).
mount | grep ${GOCRYPT_TARGET_DIR}
if [ ${?} == 0 ]; then
umount ${GOCRYPT_TARGET_DIR}
fi

GOCRYPT_CONFIG=~/.gocryptfs/backup.conf
MASTER_KEY=<master key>

# Make sure that the gocryptfs directory is mounted.
mkdir -p ${GOCRYPT_TARGET_DIR} || exit 1
modprobe fuse || exit 1
gocryptfs -config ${GOCRYPT_CONFIG} -masterkey ${MASTER_KEY} -reverse ${TARGET_DIR} ${GOCRYPT_TARGET_DIR} || exit 1

# Get the SFTP target.
TARGET_HOST=$( dig +short <sftp host> @<powerdns host> )

SSH_USERNAME=$( hostname )

# Then copy files over.
cat <<EOF | sftp -b - ${SSH_USERNAME}@${TARGET_HOST}
chdir upload
-rm *
lchdir ${GOCRYPT_TARGET_DIR}
put *
EOF

# Clean up.
umount ${GOCRYPT_TARGET_DIR}
rmdir -p ${GOCRYPT_TARGET_DIR} || true

Note that the dig command is used if your PowerDNS is not hooked up to your primary DNS (which can be quite annoying if you're using Samba as your domain controller, as the BIND9_DLZ module is not commonly provided for Samba distributions). If yours is nicely hooked up, you can just specify the SFTP host directly in the SFTP connection line.

Cloud

Centralized backups still aren't enough, though. The next step involves storing (encrypted) offsite backups in case things go horribly wrong. Fortunately, Duplicati supports multiple backup destinations (in my case, I'm using Google Drive via G Suite), is free, and has a good feature set (including a sensible smart backup retention schedule). Setting up the official Docker container within Kubernetes is fairly straightforward, as usual. First off, a read-only version of the storage above:

backup-ro.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: backup-ro-pv
labels:
   name: backup-ro-pv
spec:
capacity:
   storage: <storage>
volumeMode: Filesystem
accessModes:
  - ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
   path: <path>
nodeAffinity:
   required:
     nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
         operator: In
         values:
          - <system>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-ro-pvc
spec:
accessModes:
    - ReadOnlyMany
volumeMode: Filesystem
resources:
   requests:
     storage: <storage>
storageClassName: local-storage
selector:
   matchLabels:
     name: "backup-ro-pv"

And then the application container:

backup-duplicati.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: backup-duplicati-pv
labels:
   name: backup-duplicati-pv
spec:
capacity:
   storage: 1Gi
volumeMode: Filesystem
accessModes:
  - ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
   path: /var/data/duplicati
nodeAffinity:
   required:
     nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
         operator: In
         values:
          - <hostname>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-duplicati-pvc
spec:
accessModes:
    - ReadWriteOnce
volumeMode: Filesystem
resources:
   requests:
     storage: 1Gi
storageClassName: local-storage
selector:
   matchLabels:
     name: "backup-duplicati-pv"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-duplicati
labels:
   app: backup-duplicati
spec:
replicas: 1
selector:
   matchLabels:
     app: backup-duplicati
template:
   metadata:
     labels:
       app: backup-duplicati
   spec:
     containers:
      - name: backup-duplicati
       image: duplicati/duplicati:latest
       command: ["/usr/sbin/tini", "--"]
       args: ["/usr/bin/duplicati-server", "--webservice-port=8200", "--webservice-interface=any", "--webservice-allowed-hostnames=*"]
       volumeMounts:
        - name: backup-ro-pvc
         mountPath: <path>
        - name: backup-duplicati-pvc
         mountPath: /data
       ports:
        - containerPort: 8200
     volumes:
      - name: backup-ro-pvc
       persistentVolumeClaim:
         claimName: backup-ro-pvc
      - name: backup-duplicati-pvc
       persistentVolumeClaim:
         claimName: backup-duplicati-pvc
---
kind: Service
apiVersion: v1
metadata:
name: backup-duplicati
labels:
   app: backup-duplicati
spec:
selector:
   app: backup-duplicati
ports:
  - protocol: TCP
   port: 8200
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: backup-duplicati
labels:
   app: backup-duplicati
spec:
rules:
  - host: <hostname>
   http:
     paths:
      - path: /
       backend:
         serviceName: backup-duplicati
         servicePort: 8200

At this point, you can connect to the Duplicati hostname you specified, then follow the standard GUI documentation to set up the basics, and you're done!

(Re-)Building the world

Local

Remote

Cloud

External Links

Blog Archive

Blog Categories

Recent Blog Posts