Manually create a backup of a healthy etcd cluster and check to see if the backup was created successfully.
Backups of healthy etcd clusters can be used to restore the cluster if it becomes unhealthy at any point.
The commands in this procedure can be run on any master node (ncn-mXXX) or worker node (ncn-wXXX) on the system.
A healthy etcd cluster is available on the system. See Check the Health and Balance of etcd Clusters.
Create a backup for the desired etcd cluster.
Create variables which will be used throughout this procedure.
The example below is backing up the etcd cluster for the Boot Orchestration Service (BOS) named cray-bos-etcd-backup_DATE_TIME.
SERVICE=cray-bos
BACKUP_NAME=$SERVICE-etcd-backup_`date '+%Y-%m-%d_%H-%M-%S'`
JOB=$(kubectl exec -it -n operators \
$(kubectl get pod -n operators | grep etcd-backup-restore | head -1 | awk '{print $1}') \
-c util -- create_backup $SERVICE $BACKUP_NAME | cut -d " " -f 1); echo $JOB
Example output:
etcdbackup.etcd.database.coreos.com/cray-bos-etcd-cluster-manual-backup-25847
Check the status of the backup.
kubectl -n services get $JOB -o json | jq '.spec.s3.path, .status'
Example output:
"etcd-backup/cray-bos/cray-bos-etcd-backup_2023-03-08_20-08-07"
{
"etcdRevision": 405927,
"etcdVersion": "3.3.22",
"lastExecutionDate": "2023-03-08T19:57:42Z",
"lastSuccessDate": "2023-03-08T19:57:42Z",
"succeeded": true
}
To retrieve the created backup use the following command:
cray artifacts get etcd-backup $SERVICE/$BACKUP_NAME $BACKUP_NAME