Archive
Blog - posts for June 2020
Jun 11 2020
(Re-)Building the world
One of the things you really need with systems is backups. I've (unfortunately) been pretty bad about those too, so have taken advantage of the recent insanity to finally put together a backup routine. In this instance, the goal is to have one an onsite backup (effectively a snapshot) with offsite backups. It's a layered system, however, so requires a little bit of explanation. Note that this takes advantage of Kubernetes and the previously mentioned ExternalDNS setup for easily addressable hostnames.
Local
We want secure backups, so in this case, we're opting for gocryptfs in reverse mode in order to have straightforward secure backups, where the goals include:
- Encrypted backups where it's okay for the local system has the key laying about (since it's the system that generates the data, after all), but the central store lacks that same key.
- Reasonable space usage (i.e. doubling space requirements for encryption isn't fun).
- Reasonable package overhead (e.g. having to run an entire Java engine on a lightweight system isn't fun).
We need gocryptfs installed from edge/testing:
$ vi /etc/apk/repositories
$ apk add gocryptfs
# Disable the edge/testing repository if you're not living on it
$ vi /etc/apk/repositories
Then, set up the basics for the mount. For this, we'll be using /root/backup as our target directory:
$ gocryptfs -init -reverse backup
# Enter password here, record the master key.
$ mkdir /root/.gocryptfs
$ chmod go-rwx /root/.gocryptfs
$ mv /root/backup/.gocryptfs.reverse.conf /root/.gocryptfs/backup.conf
At this point, it's now possible to use gocryptfs' reverse mode in order to create an encrypted folder that uses no additional space:
$ modprobe fuse
$ gocryptfs -config /root/.gocryptfs/backup.conf -masterkey <master key> -reverse /root/backup /root/backup-crypt
This directory can be easily copied to another location.</p><p>But what about backup restores, you ask? The backup configuration will no longer exist. Conveniently, it's easy enough to regenerate the configuration (although you would be using the -reverse flag to generate the config). Just make sure you record the master key in a safe place.
Remote
The next step is creating a central location to store the files, where the goals include:
- Encryption in transit (mostly to protect credentials, as the backups themselves are already encrypted).
- Account isolation (i.e. one system is unable to access another system's backups).
- Minimal leakage of credentials.
In this case, SFTP (via SSH) makes the most sense, particularly once ChrootDirectory and ForceCommand internal-sftp are enabled. On the client side, sftp -b allows for basic scripting. In this instance, this SFTP Docker container within Kubernetes works out well. The first step involves setting up local storage:
kind: PersistentVolume
metadata:
name: backup-rw-pv
labels:
name: backup-rw-pv
spec:
capacity:
storage: <storage>
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: <path>
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <system>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-rw-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: <storage>
storageClassName: local-storage
selector:
matchLabels:
name: "backup-rw-pv"
Then, followed up by the actual application container:
kind: ConfigMap
metadata:
name: backup-sftp-users
data:
users.conf: |
<user entries>
---
apiVersion: v1
kind: ConfigMap
metadata:
name: backup-sftp-init
data:
init-sftp.sh: |
#!/bin/sh
cat << EOF > /etc/ssh/ssh_host_ed25519_key
-----BEGIN OPENSSH PRIVATE KEY-----
<private key>
-----END OPENSSH PRIVATE KEY-----
EOF
cat << EOF > /etc/ssh/ssh_host_ed25519_key.pub
<public key>
EOF
cat << EOF > /etc/ssh/ssh_host_rsa_key
-----BEGIN RSA PRIVATE KEY-----
<private key>
-----END RSA PRIVATE KEY-----
EOF
cat << EOF > /etc/ssh/ssh_host_rsa_key.pub
<public key>
EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-sftp
labels:
app: backup-sftp
spec:
replicas: 1
selector:
matchLabels:
app: backup-sftp
template:
metadata:
labels:
app: backup-sftp
spec:
containers:
- name: backup-sftp
image: atmoz/sftp:alpine
volumeMounts:
- name: sftp-users
mountPath: /etc/sftp
- name: sftp-init
mountPath: /etc/sftp.d
- name: backup-rw-pvc
mountPath: /home
ports:
- containerPort: 22
volumes:
- name: sftp-users
configMap:
name: backup-sftp-users
- name: sftp-init
configMap:
name: backup-sftp-init
defaultMode: 0744
- name: backup-rw-pvc
persistentVolumeClaim:
claimName: backup-rw-pvc
---
apiVersion: v1
kind: Service
metadata:
name: backup-sftp
annotations:
external-dns.alpha.kubernetes.io/hostname: <hostname>
metallb.universe.tf/address-pool: <pool>
labels:
app: backup-sftp
spec:
type: LoadBalancer
ports:
- port: 22
targetPort: 22
selector:
app: backup-sftp
This hooks up everything up neatly. The user entries follow this format:
Best practices involve leaving pass empty and using SSH keys instead. Due to having to juggle permissions appropriately, SSH keys under the <path>/<user>/.ssh/keys directory are added to the authorized_keys file, so public keys should be added there. In order to have the container recognize new users, however, the container needs to be restarted:
- Add a user entry to the YAML configuration.
- Add the SSH public key to the correct directory (setting new directory permissions appropriately).
- Ensure that Kubernetes has read the new YAML configuration.
- Restart the SFTP pod (likely by killing the current pod).
Having client systems uploading their individual backups is done via a (simple?) script, probably located in somewhere like /etc/periodic/daily (so that it's automatically run nightly):
TARGET_DIR=/var/backup
# Make local backups.
rm -rf ${TARGET_DIR}/*
<commands to generate the backup here>
GOCRYPT_TARGET_DIR=${TARGET_DIR}-crypt
# Make sure that the gocryptfs directory isn't still mounted from before (state reset).
mount | grep ${GOCRYPT_TARGET_DIR}
if [ ${?} == 0 ]; then
umount ${GOCRYPT_TARGET_DIR}
fi
GOCRYPT_CONFIG=~/.gocryptfs/backup.conf
MASTER_KEY=<master key>
# Make sure that the gocryptfs directory is mounted.
mkdir -p ${GOCRYPT_TARGET_DIR} || exit 1
modprobe fuse || exit 1
gocryptfs -config ${GOCRYPT_CONFIG} -masterkey ${MASTER_KEY} -reverse ${TARGET_DIR} ${GOCRYPT_TARGET_DIR} || exit 1
# Get the SFTP target.
TARGET_HOST=$( dig +short <sftp host> @<powerdns host> )
SSH_USERNAME=$( hostname )
# Then copy files over.
cat <<EOF | sftp -b - ${SSH_USERNAME}@${TARGET_HOST}
chdir upload
-rm *
lchdir ${GOCRYPT_TARGET_DIR}
put *
EOF
# Clean up.
umount ${GOCRYPT_TARGET_DIR}
rmdir -p ${GOCRYPT_TARGET_DIR} || true
Note that the dig command is used if your PowerDNS is not hooked up to your primary DNS (which can be quite annoying if you're using Samba as your domain controller, as the BIND9_DLZ module is not commonly provided for Samba distributions). If yours is nicely hooked up, you can just specify the SFTP host directly in the SFTP connection line.
Cloud
Centralized backups still aren't enough, though. The next step involves storing (encrypted) offsite backups in case things go horribly wrong. Fortunately, Duplicati supports multiple backup destinations (in my case, I'm using Google Drive via G Suite), is free, and has a good feature set (including a sensible smart backup retention schedule). Setting up the official Docker container within Kubernetes is fairly straightforward, as usual. First off, a read-only version of the storage above:
kind: PersistentVolume
metadata:
name: backup-ro-pv
labels:
name: backup-ro-pv
spec:
capacity:
storage: <storage>
volumeMode: Filesystem
accessModes:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: <path>
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <system>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-ro-pvc
spec:
accessModes:
- ReadOnlyMany
volumeMode: Filesystem
resources:
requests:
storage: <storage>
storageClassName: local-storage
selector:
matchLabels:
name: "backup-ro-pv"
And then the application container:
kind: PersistentVolume
metadata:
name: backup-duplicati-pv
labels:
name: backup-duplicati-pv
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /var/data/duplicati
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <hostname>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: backup-duplicati-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: local-storage
selector:
matchLabels:
name: "backup-duplicati-pv"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-duplicati
labels:
app: backup-duplicati
spec:
replicas: 1
selector:
matchLabels:
app: backup-duplicati
template:
metadata:
labels:
app: backup-duplicati
spec:
containers:
- name: backup-duplicati
image: duplicati/duplicati:latest
command: ["/usr/sbin/tini", "--"]
args: ["/usr/bin/duplicati-server", "--webservice-port=8200", "--webservice-interface=any", "--webservice-allowed-hostnames=*"]
volumeMounts:
- name: backup-ro-pvc
mountPath: <path>
- name: backup-duplicati-pvc
mountPath: /data
ports:
- containerPort: 8200
volumes:
- name: backup-ro-pvc
persistentVolumeClaim:
claimName: backup-ro-pvc
- name: backup-duplicati-pvc
persistentVolumeClaim:
claimName: backup-duplicati-pvc
---
kind: Service
apiVersion: v1
metadata:
name: backup-duplicati
labels:
app: backup-duplicati
spec:
selector:
app: backup-duplicati
ports:
- protocol: TCP
port: 8200
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: backup-duplicati
labels:
app: backup-duplicati
spec:
rules:
- host: <hostname>
http:
paths:
- path: /
backend:
serviceName: backup-duplicati
servicePort: 8200
At this point, you can connect to the Duplicati hostname you specified, then follow the standard GUI documentation to set up the basics, and you're done!