Keycloak Service Recovery

The following covers redeploying the Keycloak service and restoring the data.

Prerequisites

  • The system is fully installed and has transitioned off of the LiveCD.
  • All activities required for site maintenance are complete.
  • A backup or export of the data already exists.
  • The latest CSM documentation has been installed on the master nodes. See Check for Latest Documentation.
  • The Cray CLI is configured on the node where the procedure is being performed. See Configure the Cray CLI.

Service recovery for Keycloak

  1. (ncn-mw#) Verify that a backup of the Keycloak Postgres data exists.

    1. Set and export the CRAY_CREDENTIALS environment variable.

      This will permit simple CLI operations that are needed for the command in the next step. See Authenticate an Account with the Command Line.

    2. List the Postgres logical backups by date.

      cray artifacts list postgres-backup --format json | jq -r '.artifacts[] | select(.Key | contains("spilo/keycloak")) | "\(.LastModified) \(.Key)"'

      Example output:

      2023-03-23T02:10:11.158000+00:00 spilo/keycloak-postgres/ed8f6691-9da7-4662-aa67-9c786fa961ee/logical_backups/1679537409.sql.gz
2023-03-24T02:10:12.689000+00:00 spilo/keycloak-postgres/ed8f6691-9da7-4662-aa67-9c786fa961ee/logical_backups/1679623811.sql.gz

    3. Unset the CRAY_CREDENTIALS environment variable and remove the temporary token file.

      unset CRAY_CREDENTIALS
rm -v /tmp/setup-token.json

  2. (ncn-mw#) Uninstall the chart and wait for the resources to terminate.

    1. Uninstall the chart.

      helm uninstall -n services cray-keycloak

      Example output:

      release "cray-keycloak" uninstalled

    2. Wait for the resources to terminate.

      watch "kubectl get pods -n services | grep keycloak | grep -v 'keycloak-users-localize\|keycloak-vcs-user\|logical-backup'"

    Example output:

    No resources found in services namespace.

  3. (ncn-mw#) Redeploy the chart and wait for the resources to start.

    Follow the Redeploying a Chart procedure with the following specifications:

    • Chart name: cray-keycloak

    • Base manifest name: platform

    • When reaching the step to update customizations, no edits need to be made to the customizations file.

    • When reaching the step to validate that the redeploy was successful, perform the following step:

      Only follow this step as part of the previously linked chart redeploy procedure.

      Wait for the resources to start.

      watch "kubectl get pods -n services | grep keycloak"

      Example output:

      cray-keycloak-0                                                   2/2     Running     0          32m
cray-keycloak-1                                                   2/2     Running     0          32m
cray-keycloak-2                                                   2/2     Running     0          32m
keycloak-postgres-0                                               3/3     Running     0          32m
keycloak-postgres-1                                               3/3     Running     0          31m
keycloak-postgres-2                                               3/3     Running     0          30m
keycloak-setup-1-9kdl2                                            0/2     Completed   0          32m
keycloak-users-localize-1-jjb9b                                   2/2     Running     0          32m
keycloak-vcs-user-1-gqftw                                         0/2     Completed   0          31m
keycloak-wait-for-postgres-1-xt4nv                                0/2     Completed   0          32m

  4. (ncn-mw#) Restore the critical data.

    See Restore Postgres for Keycloak.