Version Control Service (VCS)

VCS web interface

The Version Control Service (VCS) includes a web interface for repository management, pull requests, and a visual view of all repositories and organizations. The following URL is for the VCS web interface:

https://vcs.ext.system.domain.com

(ncn-mw#) To look up the external system domain name, run the following command:

kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | base64 -d | grep "external:"

Example output:

      external: ext.system.domain.com

Cloning a VCS repository

On cluster nodes, the VCS service can be accessed through the gateway. VCS credentials for the crayvcs user are required before cloning a repository (see VCS Administrative User).

(ncn#) To clone a repository in the cray organization, use the following command:

git clone https://api-gw-service-nmn.local/vcs/cray/REPO_NAME.git

Access the cray Gitea organization

The VCS UI uses Keycloak to authenticate users on the system. However, users from external authentication sources are not automatically associated with permissions in the cray Gitea organization. As a result, users configured via Keycloak can log in and create organizations and repositories of their own, but they cannot modify the cray organization that is created during system installation, unless they are given permissions to do so.

The crayvcs Gitea admin user that is created during CSM installation can log in to the UI via Keycloak. To allow users other than crayvcs to have access to repositories in the cray organization, use the following procedure:

  1. Log in to VCS as the crayvcs user on the system.

    URL: https://vcs.SHASTA_CLUSTER_DNS_NAME

  2. Navigate to the cray organization owners page.

    URL: https://vcs.SHASTA_CLUSTER_DNS_NAME/vcs/cray/teams/owners

  3. Enter the username of the user who should have access to the organization in the Search user... text field, and click the Add Team Member button.

IMPORTANT The Owner role has full access to all repositories in the organization, as well as administrative access to the organization, including the ability to create and delete repositories.

For granting non-administrative access to the organization and its repositories, create a new team at the following URL:

https://vcs.SHASTA_CLUSTER_DNS_NAME/vcs/org/cray/teams/new

Select the permissions appropriately, and then navigate to the following URL to add members to the newly created team:

https://vcs.SHASTA_CLUSTER_DNS_NAME/vcs/org/cray/teams/NEWTEAM

Backup and restore data

Data for Gitea is stored in two places: Git content is stored directly in a PVC, while structural data, such as Gitea users and the list and attributes of repositories, is stored in a Postgres database. Because of this, both sources must be backed up and restored together.

Automated backup and restore

Automated backup

(ncn-mw#) Running the following script creates a tar archive containing both the Postgres and PVC data.

The argument to the script is the directory where the resulting archive should be created.

/usr/share/doc/csm/scripts/operations/configuration/backup_vcs.sh /root

The end of the output will include the path to the backup archive. For example:

Gitea/VCS data successfully backed up to /root/gitea-vcs-20240626192742-dRW95b.tgz

Be sure to save the resulting archive file to a safe location.

Automated restore

(ncn-mw#) The archive generated by the Automated backup script can be used as input to the following automated restore script.

/usr/share/doc/csm/scripts/operations/configuration/restore_vcs.sh /root/gitea-vcs-20240626192742-dRW95b.tgz

Manual backup and restore

Manually backup Postgres data

  1. (ncn-mw#) Determine which Postgres member is the leader.

    kubectl exec gitea-vcs-postgres-0 -n services -c postgres -it -- patronictl list

    Example output:

    + Cluster: gitea-vcs-postgres (6995618180238446669) -----+----+-----------+
|        Member        |     Host     |  Role  |  State  | TL | Lag in MB |
+----------------------+--------------+--------+---------+----+-----------+
| gitea-vcs-postgres-0 |  10.45.0.21  | Leader | running |  1 |           |
| gitea-vcs-postgres-1 | 10.46.128.19 |        | running |  1 |         0 |
| gitea-vcs-postgres-2 |  10.47.0.21  |        | running |  1 |         0 |
+----------------------+--------------+--------+---------+----+-----------+

  2. (ncn-mw#) Log into the leader pod and dump the data to a local file.

    POSTGRES_LEADER=gitea-vcs-postgres-0
kubectl exec -it ${POSTGRES_LEADER} -n services -c postgres -- pg_dumpall -c -U postgres > gitea-vcs-postgres.sql

  3. (ncn-mw#) Determine what secrets are associated with the PostgreSQL credentials:

    kubectl get secrets -n services | grep gitea-vcs-postgres.credentials

    Example output:

    postgres.gitea-vcs-postgres.credentials                   Opaque                                2      13d
service-account.gitea-vcs-postgres.credentials            Opaque                                2      13d
standby.gitea-vcs-postgres.credentials                    Opaque                                2      13d

  4. (ncn-mw#) Export each secret to a manifest file.

    The creationTimestamp, resourceVersion, selfLink, and uid metadata fields are removed from each secret as they are exported.

    SECRETS="postgres service-account standby"
tmpfile=$(mktemp)
echo "---" > gitea-vcs-postgres.manifest
for secret in $SECRETS; do
    kubectl get secret "${secret}.gitea-vcs-postgres.credentials" -n services -o yaml > "${tmpfile}"
    for field in creationTimestamp resourceVersion selfLink uid; do
        yq d -i "${tmpfile}" "metadata.${field}"
    done
    cat "${tmpfile}" >> gitea-vcs-postgres.manifest
    echo "---" >> gitea-vcs-postgres.manifest
done
rm "${tmpfile}"

  5. Copy all files to a safe location.

Manually backup PVC data

(ncn-mw#) The VCS Postgres backups should be accompanied by backups of the VCS PVC. The export process can be run at any time while the service is running using the following commands:

POD=$(kubectl -n services get pod -l app.kubernetes.io/instance=gitea -o json | jq -r '.items[] | .metadata.name')
kubectl -n services exec ${POD} -- tar -cvf /tmp/vcs.tar /var/lib/gitea/
kubectl -n services cp ${POD}:/tmp/vcs.tar ./vcs.tar

Be sure to save the resulting tar file to a safe location.

Manually restore Postgres data

See Restore Postgres for VCS.

Manually restore PVC data

(ncn-mw#) When restoring the VCS Postgres database, the PVC should also be restored to the same point in time. The restore process can be run at any time while the service is running using the following commands:

POD=$(kubectl -n services get pod -l app.kubernetes.io/instance=gitea -o json | jq -r '.items[] | .metadata.name')
kubectl -n services cp ./vcs.tar ${POD}:/tmp/vcs.tar
kubectl -n services exec ${POD} -- tar -C / -xvf /tmp/vcs.tar
kubectl -n services rollout restart deployment gitea-vcs

Alternative backup/restore strategy

An alternative to the separate backups of the Postgres and PVC data is to backup the Git data. This has the advantage that only one backup is needed and that the Git backups can be imported into any Git server, not just Gitea. This has the disadvantage that some information about the Gitea deployment is lost (such as user and organization information) and may need to be recreated manually if the VCS deployment is lost.

The following scripts create and use a vcs-content directory that contains all Git data. This should be copied to a safe location after export, and moved back to the system before import.

Alternative export method

WARNING: The following example uses the VCS admin username and password in plaintext on the command line, meaning it will be stored in the shell history as well as be visible to all users on the system in the process table. These dangers can be avoided by modifying or replacing the curl command (such as supplying the credentials to curl using the --netrc-file argument instead of the --user argument, or replacing it with a simple Python script).

(ncn-mw#) Use the following commands to do the export:

RESULTS=vcs-content
mkdir $RESULTS
VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
git config --global credential.helper store
echo "https://${VCS_USER}:${VCS_PASSWORD}@api-gw-service-nmn.local" > ~/.git-credentials
for repo in $(curl -s https://api-gw-service-nmn.local/vcs/api/v1/orgs/cray/repos --user ${VCS_USER}:${VCS_PASSWORD}| jq -r '.[] | .name') ; do
    git clone --mirror https://api-gw-service-nmn.local/vcs/cray/${repo}.git
    cd ${repo}.git
    git bundle create ${repo}.bundle --all
    cp ${repo}.bundle ../$RESULTS
    cd ..
    rm -r $repo.git
done

Alternative import method

(ncn-mw#) Use the following commands to do the import:

SOURCE=vcs-content
VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
git config --global credential.helper store
echo "https://${VCS_USER}:${VCS_PASSWORD}@api-gw-service-nmn.local" > ~/.git-credentials
for file in $(ls $SOURCE); do
    repo=$(echo $file | sed 's/.bundle$//')
    git clone --mirror ${SOURCE}/${repo}.bundle
    cd ${repo}.git
    git remote set-url origin https://api-gw-service-nmn.local/vcs/cray/${repo}.git
    git push
    cd ..
    rm -r ${repo}.git
done

Prior to import, the repository structure may need to be recreated if it has not already been by an install.

Adjust the repository list as necessary, if any additional repositories are present. Repository settings such as public or private will also need to be manually set, if applicable.

(ncn-mw#) For example:

WARNING: The following example uses the VCS admin username and password in plaintext on the command line, meaning it will be stored in the shell history as well as be visible to all users on the system in the process table. These dangers can be avoided by modifying or replacing the curl command (such as supplying the credentials to curl using the --netrc-file argument instead of the --user argument, or replacing it with a simple Python script).

VCS_USER=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_username}} | base64 --decode)
VCS_PASSWORD=$(kubectl get secret -n services vcs-user-credentials --template={{.data.vcs_password}} | base64 --decode)
REPOS="analytics-config-management cos-config-management cpe-config-management slurm-config-management sma-config-management uan-config-management csm-config-management"
for repo in $REPOS ; do
    curl -X POST https://api-gw-service-nmn.local/vcs/api/v1/orgs/cray/repos -u ${VCS_USER}:${VCS_PASSWORD} -d name=${repo}
done