Backup and Restore VCS Data

This document describes how to perform backups and restores of the Version Control Service (VCS) data. For general information, see Version Control Service (VCS).

Overview

Data for Gitea is stored in two places: Git content is stored directly in a PVC, while structural data, such as Gitea users and the list and attributes of repositories, is stored in a Postgres database. Because of this, both sources must be backed up and restored together. A full restore is not possible without backups of both the PVC and Postgres data. CSM performs automatic periodic backups of the Postgres data, but the PVC data is not automatically backed up by CSM. Administrators are recommended to take periodic backups, using one of the methods outlined on this page.

Automated backup and restore

This is the recommended method for performing backups and restores of VCS data.

Automated backup

(ncn-mw#) Running the following script creates a tar archive containing both the Postgres and PVC data.

The argument to the script is the directory where the resulting archive should be created.

/usr/share/doc/csm/scripts/operations/configuration/backup_vcs.sh /root

The end of the output will include the path to the backup archive. For example:

Gitea/VCS data successfully backed up to /root/gitea-vcs-20240626192742-dRW95b.tgz

Be sure to save the resulting archive file to a safe location.

Automated restore

(ncn-mw#) The archive generated by the Automated backup script can be used as input to the following automated restore script.

/usr/share/doc/csm/scripts/operations/configuration/restore_vcs.sh /root/gitea-vcs-20240626192742-dRW95b.tgz

Manual backup and restore

Manual backup procedure

Remember that backups of both the Postgres and PVC data are necessary in order to perform a complete restore of VCS data.

  1. Manually backup Postgres data
  2. Manually backup PVC data

1. Manually backup Postgres data

  1. (ncn-mw#) Determine which Postgres member is the leader.

    kubectl exec gitea-vcs-postgres-0 -n services -c postgres -it -- patronictl list
    

    Example output:

    + Cluster: gitea-vcs-postgres (6995618180238446669) -----+----+-----------+
    |        Member        |     Host     |  Role  |  State  | TL | Lag in MB |
    +----------------------+--------------+--------+---------+----+-----------+
    | gitea-vcs-postgres-0 |  10.45.0.21  | Leader | running |  1 |           |
    | gitea-vcs-postgres-1 | 10.46.128.19 |        | running |  1 |         0 |
    | gitea-vcs-postgres-2 |  10.47.0.21  |        | running |  1 |         0 |
    +----------------------+--------------+--------+---------+----+-----------+
    
  2. (ncn-mw#) Log into the leader pod and dump the data to a local file.

    POSTGRES_LEADER=gitea-vcs-postgres-0
    kubectl exec -it ${POSTGRES_LEADER} -n services -c postgres -- pg_dumpall -c -U postgres > gitea-vcs-postgres.sql
    
  3. (ncn-mw#) Determine what secrets are associated with the PostgreSQL credentials:

    kubectl get secrets -n services | grep gitea-vcs-postgres.credentials
    

    Example output:

    postgres.gitea-vcs-postgres.credentials                   Opaque                                2      13d
    service-account.gitea-vcs-postgres.credentials            Opaque                                2      13d
    standby.gitea-vcs-postgres.credentials                    Opaque                                2      13d
    
  4. (ncn-mw#) Export each secret to a manifest file.

    The creationTimestamp, resourceVersion, selfLink, and uid metadata fields are removed from each secret as they are exported.

    SECRETS="postgres service-account standby"
    tmpfile=$(mktemp)
    echo "---" > gitea-vcs-postgres.manifest
    for secret in $SECRETS; do
        kubectl get secret "${secret}.gitea-vcs-postgres.credentials" -n services -o yaml > "${tmpfile}"
        for field in creationTimestamp resourceVersion selfLink uid; do
            yq d -i "${tmpfile}" "metadata.${field}"
        done
        cat "${tmpfile}" >> gitea-vcs-postgres.manifest
        echo "---" >> gitea-vcs-postgres.manifest
    done
    rm "${tmpfile}"
    
  5. Copy all files to a safe location.

2. Manually backup PVC data

(ncn-mw#) The VCS Postgres backups should be accompanied by backups of the VCS PVC. The export process can be run at any time while the service is running using the following commands:

POD=$(kubectl -n services get pod -l app.kubernetes.io/instance=gitea -o json | jq -r '.items[] | .metadata.name')
kubectl -n services exec ${POD} -- tar -cvf /tmp/vcs.tar /var/lib/gitea/
kubectl -n services cp ${POD}:/tmp/vcs.tar ./vcs.tar

Be sure to save the resulting tar file to a safe location.

Manual restore procedure

In order to completely restore the VCS data, first the Postgres data must be restored, then the PVC data must be restored.

1. Manually restore Postgres data

  1. (ncn-mw#) Copy the database dump to an accessible location.

    • If a manual dump of the database was taken, then check that the dump file exists in a location off the Postgres cluster. It will be needed in the steps below.
    • If the database is being automatically backed up, then the most recent version of the dump and the secrets should exist in the postgres-backup S3 bucket. These will be needed in the steps below. List the files in the postgres-backup S3 bucket and if the files exist, download the dump and secrets out of the S3 bucket. The cray artifacts CLI can be used to list and download the files. Note that the .psql file contains the database dump and the .manifest file contains the secrets. The CLI must be configured and authenticated in order to do this. See Configure the Cray CLI.
    1. List the available backups.

      cray artifacts list postgres-backup --format json | jq -r '.artifacts[] | select(.Key | contains("spilo/gitea")) | "\(.LastModified) \(.Key)"'
      

      Example output:

      2023-03-22T01:10:26.475500+00:00 spilo/gitea-vcs-postgres/9b8df946-ef39-4880-86a7-f8c21b71c542/logical_backups/1679447424.sql.gz
      2023-03-23T01:10:26.395000+00:00 spilo/gitea-vcs-postgres/9b8df946-ef39-4880-86a7-f8c21b71c542/logical_backups/1688548424.sql.gz
      
    2. Set the environment variables to the name of the backup file.

      BACKUP=spilo/gitea-vcs-postgres/9b8df946-ef39-4880-86a7-f8c21b71c542/logical_backups/1688548424.sql.gz
      DUMPFILE=$(basename ${BACKUP})
      
    3. Download the backup files.

      cray artifacts get postgres-backup "${BACKUP}" "./${DUMPFILE}"
      
  2. (ncn-mw#) Set helper variables.

    SERVICE=gitea-vcs
    SERVICELABEL=vcs
    NAMESPACE=services
    POSTGRESQL=gitea-vcs-postgres
    
  3. (ncn-mw#) Scale the VCS service to 0.

    kubectl scale deployment ${SERVICE} -n "${NAMESPACE}" --replicas=0
    
  4. (ncn-mw#) Wait for the pods to terminate.

    while kubectl get pods -n "${NAMESPACE}" -l app.kubernetes.io/name="${SERVICELABEL}" | grep -qv NAME ; do
        echo "  waiting for pods to terminate"; sleep 2
    done
    
  5. (ncn-mw#) Delete the VCS Postgres cluster.

    kubectl get postgresql "${POSTGRESQL}" -n "${NAMESPACE}" -o json | jq 'del(.spec.selector)' |
        jq 'del(.spec.template.metadata.labels."controller-uid")' | jq 'del(.status)' > postgres-cr.json
    kubectl delete -f postgres-cr.json
    
  6. (ncn-mw#) Wait for the pods to terminate.

    while kubectl get pods -l "application=spilo,cluster-name=${POSTGRESQL}" -n "${NAMESPACE}" | grep -qv NAME ; do
        echo "  waiting for pods to terminate"; sleep 2
    done
    
  7. (ncn-mw#) Create a new single instance VCS Postgres cluster.

    cp -v postgres-cr.json postgres-orig-cr.json
    jq '.spec.numberOfInstances = 1' postgres-orig-cr.json > postgres-cr.json
    kubectl create -f postgres-cr.json
    
  8. (ncn-mw#) Wait for the pod to start.

    while ! kubectl get pods -l "application=spilo,cluster-name=${POSTGRESQL}" -n "${NAMESPACE}" | grep -qv NAME ; do
        echo "  waiting for pod to start"; sleep 2
    done
    
  9. (ncn-mw#) Wait for the Postgres cluster to start running.

    while [ $(kubectl get postgresql "${POSTGRESQL}" -n "${NAMESPACE}" -o json | jq -r '.status.PostgresClusterStatus') != "Running" ]
    do
        echo "  waiting for postgresql to start running"; sleep 2
    done
    
  10. (ncn-mw#) Copy the database dump file to the Postgres member.

    kubectl cp "./${DUMPFILE}" "${POSTGRESQL}-0:/home/postgres/${DUMPFILE}" -c postgres -n "${NAMESPACE}" 
    
  11. (ncn-mw#) Restore the data.

    kubectl exec "${POSTGRESQL}-0" -c postgres -n "${NAMESPACE}" -it -- bash -c "zcat -f ${DUMPFILE} | psql -U postgres" 
    

    Errors such as ... already exists can be ignored; the restore can be considered successful when it completes.

  12. Update the gitea-vcs-postgres secrets in Postgres.

    1. (ncn-mw#) From the three gitea-vcs-postgres secrets, collect the password for each Postgres username: postgres, service_account, and standby.

      for secret in postgres.gitea-vcs-postgres.credentials service-account.gitea-vcs-postgres.credentials \
          standby.gitea-vcs-postgres.credentials
      do
          echo -n "secret ${secret} username & password: "
          echo -n "`kubectl get secret "${secret}" -n "${NAMESPACE}" -ojsonpath='{.data.username}' | base64 -d` "
          echo `kubectl get secret "${secret}" -n "${NAMESPACE}" -ojsonpath='{.data.password}'| base64 -d`
      done
      

      Example output:

      secret postgres.gitea-vcs-postgres.credentials username & password: postgres ABCXYZ
      secret service-account.gitea-vcs-postgres.credentials username & password: service_account ABC123
      secret standby.gitea-vcs-postgres.credentials username & password: standby 123456
      
    2. (ncn-mw#) kubectl exec into the Postgres pod.

      kubectl exec "${POSTGRESQL}-0" -n "${NAMESPACE}" -c postgres -it -- bash
      
    3. (pod#) Open a Postgres console.

      /usr/bin/psql postgres postgres
      
    4. (postgres#) Update the password for each user to match the values found in the secrets.

      1. Update the password for the postgres user.

        ALTER USER postgres WITH PASSWORD 'ABCXYZ';
        

        Example of successful output:

        ALTER ROLE
        
      2. Update the password for the service_account user.

        ALTER USER service_account WITH PASSWORD 'ABC123';
        

        Example of successful output:

        ALTER ROLE
        
      3. Update the password for the standby user.

        ALTER USER standby WITH PASSWORD '123456';
        

        Example of successful output:

        ALTER ROLE
        
    5. (postgres#) Exit the Postgres console with the \q command.

    6. (pod#) Exit the Postgres pod with the exit command.

  13. (ncn-mw#) Restart the Postgres cluster.

    kubectl delete pod -n "${NAMESPACE}" "${POSTGRESQL}-0"
    
  14. (ncn-mw#) Wait for the postgresql pod to start.

    while ! kubectl get pods -l "application=spilo,cluster-name=${POSTGRESQL}" -n "${NAMESPACE}" | grep -qv NAME ; do
        echo "  waiting for pods to start"; sleep 2
    done
    
  15. (ncn-mw#) Scale the Postgres cluster back to 3 instances.

    kubectl patch postgresql "${POSTGRESQL}" -n "${NAMESPACE}" --type='json' \
        -p='[{"op" : "replace", "path":"/spec/numberOfInstances", "value" : 3}]'
    
  16. (ncn-mw#) Wait for the postgresql cluster to start running.

    while [ $(kubectl get postgresql "${POSTGRESQL}" -n "${NAMESPACE}" -o json | jq -r '.status.PostgresClusterStatus') != "Running" ]
    do
        echo "  waiting for postgresql to start running"; sleep 2
    done
    
  17. (ncn-mw#) Scale the Gitea service back up.

    kubectl scale deployment ${SERVICE} -n "${NAMESPACE}" --replicas=1
    
  18. (ncn-mw#) Wait for the Gitea pods to start.

    while ! kubectl get pods -n "${NAMESPACE}" -l app.kubernetes.io/name="${SERVICELABEL}" | grep -qv NAME ; do
        echo "  waiting for pods to start"; sleep 2
    done
    

2. Manually restore PVC data

(ncn-mw#) When restoring the VCS Postgres database, the PVC should also be restored to the same point in time. The restore process can be run at any time while the service is running using the following commands:

POD=$(kubectl -n services get pod -l app.kubernetes.io/instance=gitea -o json | jq -r '.items[] | .metadata.name')
kubectl -n services cp ./vcs.tar ${POD}:/tmp/vcs.tar
kubectl -n services exec ${POD} -- tar -C / -xvf /tmp/vcs.tar
kubectl -n services rollout restart deployment gitea-vcs

Alternative backup/restore strategy

An alternative to the separate backups of the Postgres and PVC data is to backup the Git data. This has the advantage that only one backup is needed and that the Git backups can be imported into any Git server, not just Gitea. This has the disadvantage that some information about the Gitea deployment is lost (such as user and organization information) and may need to be recreated manually if the VCS deployment is lost.

The following scripts create and use a vcs-content directory that contains all Git data. This should be copied to a safe location after export, and moved back to the system before import.

Alternative export method

WARNING: The following example uses the VCS admin username and password in plaintext on the command line, meaning it will be stored in the shell history as well as be visible to all users on the system in the process table. These dangers can be avoided by modifying or replacing the curl command (such as supplying the credentials to curl using the --netrc-file argument instead of the --user argument, or replacing it with a simple Python script).

(ncn-mw#) Use the following commands to do the export:

RESULTS=vcs-content
mkdir $RESULTS
VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
git config --global credential.helper store
echo "https://${VCS_USER}:${VCS_PASSWORD}@api-gw-service-nmn.local" > ~/.git-credentials
for repo in $(curl -s https://api-gw-service-nmn.local/vcs/api/v1/orgs/cray/repos --user ${VCS_USER}:${VCS_PASSWORD}| jq -r '.[] | .name') ; do
    git clone --mirror https://api-gw-service-nmn.local/vcs/cray/${repo}.git
    cd ${repo}.git
    git bundle create ${repo}.bundle --all
    cp ${repo}.bundle ../$RESULTS
    cd ..
    rm -r $repo.git
done

Alternative import method

(ncn-mw#) Use the following commands to do the import:

SOURCE=vcs-content
VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
git config --global credential.helper store
echo "https://${VCS_USER}:${VCS_PASSWORD}@api-gw-service-nmn.local" > ~/.git-credentials
for file in $(ls $SOURCE); do
    repo=$(echo $file | sed 's/.bundle$//')
    git clone --mirror ${SOURCE}/${repo}.bundle
    cd ${repo}.git
    git remote set-url origin https://api-gw-service-nmn.local/vcs/cray/${repo}.git
    git push
    cd ..
    rm -r ${repo}.git
done

Prior to import, the repository structure may need to be recreated if it has not already been by an install.

Adjust the repository list as necessary, if any additional repositories are present. Repository settings such as public or private will also need to be manually set, if applicable.

(ncn-mw#) For example:

WARNING: The following example uses the VCS admin username and password in plaintext on the command line, meaning it will be stored in the shell history as well as be visible to all users on the system in the process table. These dangers can be avoided by modifying or replacing the curl command (such as supplying the credentials to curl using the --netrc-file argument instead of the --user argument, or replacing it with a simple Python script).

VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
REPOS="analytics-config-management uss-config-management cpe-config-management slurm-config-management sma-config-management uan-config-management csm-config-management"
for repo in $REPOS ; do
    curl -X POST https://api-gw-service-nmn.local/vcs/api/v1/orgs/cray/repos -u ${VCS_USER}:${VCS_PASSWORD} -d name=${repo}
done