Cray System Management Documentation > Cray System Management (CSM) Administration Guide > security and authentication > Manage Sealed Secrets

Manage Sealed Secrets

Sealed secrets are essential for managing sensitive information on the system. The following procedures for managing sealed secrets are included in this section:

In the following sections, the term “tracked sealed secrets” is used to describe any existing secrets stored in spec.kubernetes.tracked_sealed_secrets that are available to be regenerated.

Many of the examples in this section assume that the system was installed using a USB PIT node, and that the USB stick used to install the system is still available. If site-init is no longer available on the USB stick and a backup has not been made, a new site-init will need to be created following step 1 in the Generate Sealed Secrets Post-Install section.

The customizations.yaml file used in this procedure will be in one of the following locations depending on the state of the system:

Fresh install location: /mnt/pitdata/${CSM_DISTDIR}/shasta-cfg/customizations.yaml
Post-install location: /root/site-init/${CSM_DISTDIR}/shasta-cfg/customizations.yaml

Generate Sealed Secrets Post-Install

Sealed secrets are stored in customizations.yaml as SealedSecret resources (encrypted secrets), which are deployed by specific charts and decrypted by the sealed secrets operator. First, those secrets must be seeded, generated, and encrypted.

The steps in this section assume that the system was not installed using a USB PIT node, or that the USB stick is no longer available.

If LDAP user federation is required, then refer to Add LDAP User Federation.

Prepare to customize the customizations.yaml file.

If the customizations.yaml file is managed in an external Git repository (as recommended), then clone a local working tree. Replace the <URL> value in the following command before running it.
```
git clone <URL> /root/site-init
cd /root/site-init
```
If there is not a backup of site-init, perform the following steps to create a new one using the values stored in the Kubernetes cluster.
1. Set the CSM_DISTDIR variable to the path to the unpacked CSM release tarball.
  
  See Download and extract CSM product release.
2. Create a new site-init directory using the CSM tarball.
```
cp -r ${CSM_DISTDIR}/shasta-cfg/* /root/site-init
cd /root/site-init
```
3. Extract customizations.yaml from the site-init secret.
```
kubectl -n loftsman get secret site-init -o jsonpath='{.data.customizations\.yaml}' | base64 -d - > customizations.yaml
```
4. Extract the certificate and key used to create the sealed secrets.
```
mkdir -p certs
kubectl -n kube-system get secret sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d - > certs/sealed_secrets.crt
kubectl -n kube-system get secret sealed-secrets-key -o jsonpath='{.data.tls\.key}' | base64 -d - > certs/sealed_secrets.key
```
(Optional) Prevent tracked sealed secrets from being regenerated.

Remove the sealed secrets not being regenerated from the spec.kubernetes.tracked_sealed_secrets list in /root/site-init/${CSM_DISTDIR}/shasta-cfg/customizations.yaml prior to executing the remaining steps in this section.

Retain the REDS/MEDS/RTS credentials.
```
yq delete -i customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_reds_credentials
yq delete -i customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_meds_credentials
yq delete -i customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_hms_rts_credentials
```

Prepare to generate sealed secrets.

./utils/secrets-reencrypt.sh customizations.yaml ./certs/sealed_secrets.key ./certs/sealed_secrets.crt

Encrypt the static values in the customizations.yaml file after making changes.

The following command must be run within the site-init directory.

./utils/secrets-seed-customizations.sh customizations.yaml

Expected output looks similar to:

Creating Sealed Secret keycloak-certs
Generating type static_b64...
Creating Sealed Secret keycloak-master-admin-auth
Generating type static...
Generating type static...
Generating type randstr...
Generating type static...
Creating Sealed Secret cray_reds_credentials
Generating type static...
Generating type static...
Creating Sealed Secret cray_meds_credentials
Generating type static...
Creating Sealed Secret cray_hms_rts_credentials
Generating type static...
Generating type static...
Creating Sealed Secret vcs-user-credentials
Generating type randstr...
Generating type static...
Creating Sealed Secret generated-platform-ca-1
Generating type platform_ca...
Creating Sealed Secret pals-config
Generating type zmq_curve...
Generating type zmq_curve...
Creating Sealed Secret munge-secret
Generating type randstr...
Creating Sealed Secret slurmdb-secret
Generating type static...
Generating type static...
Generating type randstr...
Generating type randstr...
Creating Sealed Secret keycloak-users-localize
Generating type static...

Prevent Regeneration of Tracked Sealed Secrets

Before performing the task to generate or regenerate sealed secrets, administrators are able to prevent existing tracked sealed secrets from being regenerated.

To prevent regeneration, sealed secrets MUST BE REMOVED from the spec.kubernetes.tracked_sealed_secrets list in the customizations.yaml file prior to executing the “Generate Sealed Secrets” section of the Prepare Site Init procedure.

To retain the REDS/MEDS/RTS credentials:

yq delete -i /mnt/pitdata/${CSM_DISTDIR}/shasta-cfg/customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_reds_credentials
yq delete -i /mnt/pitdata/${CSM_DISTDIR}/shasta-cfg/customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_meds_credentials
yq delete -i /mnt/pitdata/${CSM_DISTDIR}/shasta-cfg/customizations.yaml spec.kubernetes.tracked_sealed_secrets.cray_hms_rts_credentials

View Tracked Sealed Secrets

Tracked sealed secrets are regenerated every time secrets are seeded (see the use of utils/secrets-seed-customizations.sh above).

To view currently tracked sealed secrets:

yq read ${CSM_PATH}/shasta-cfg/customizations.yaml spec.kubernetes.tracked_sealed_secrets

Expected output looks similar to the following:

- cray_reds_credentials
- cray_meds_credentials
- cray_hms_rts_credentials

Decrypt Sealed Secrets for Review

Use the secrets-decrypt.sh utility in the SHASTA-CFG to decrypt and review previously encrypted sealed secrets.

Syntax: secret-decrypt.sh SEALED-SECRET-NAME SEALED-SECRET-PRIVATE-KEY CUSTOMIZATIONS-YAML

For example:

cd /mnt/pitdata/prep/site-init &&
     ./utils/secrets-decrypt.sh cray_meds_credentials ./certs/sealed_secrets.key ./customizations.yaml | \
        jq .data.vault_redfish_defaults | sed -e 's/"//g' | base64 -d; echo

Expected output looks similar to the following:

{"Username": "root", "Password": "..."}

Fix an Incorrect Value in a Sealed Secret

This procedure describes how to correct an invalid password in the customizations.yaml file during an install. In the following example, a typo was made in the SNMPAuthPassword field of the vault_switch_defaults credentials in the cray_reds_credentials secret, resulting in hardware not being discovered.

The general process outlined in the following steps can be followed if a different value is incorrect.

./utils/secrets-decrypt.sh cray_reds_credentials | jq -r '.data.vault_switch_defaults' | base64 --decode

Output looks similar to the following:

{"SNMPUsername": "<USERID>", "SNMPAuthPassword": "<A-PASS>", "SNMPPrivPassword": "<P-PASS>"}

Decrypt the cray_reds_credentials secret.

./utils/secrets-decrypt.sh cray_reds_credentials > cray_reds_credentials.json

The output file should look similar to the following. Note that the data values are base64 encoded.

{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
   "name": "cray-reds-credentials",
   "creationTimestamp": null,
   "annotations": {
      "sealedsecrets.bitnami.com/cluster-wide": "true"
   },
   "ownerReferences": [
      {
      "apiVersion": "bitnami.com/v1alpha1",
      "kind": "SealedSecret",
      "name": "cray-reds-credentials",
      "uid": "",
      "controller": true
      }
   ]
},
"data": {
   "vault_redfish_defaults": "eyJDcmF5IjogeyJVc2VybmFtZSI6ICJyb290IiwgIlBhc3N3b3JkIjogImluaXRpYWwwIn19",
   "vault_switch_defaults": "eyJTTk1QVXNlcm5hbWUiOiAidGVzdHVzZXIiLCAiU05NUEF1dGhQYXNzd29yZCI6ICJ0ZXN0cGFzMSIsICJTTk1QUHJpdlBhc3N3b3JkIjogInRlc3RwYXNzMiJ9"
}
}

Decode the vault_switch_defaults credentials to a working file.

jq -r '.data.vault_switch_defaults' cray_reds_credentials.json | base64 --decode > vault_switch_defaults.json

Correct the password in the vault_switch_defaults.json file.

{"SNMPUsername": "<USERID>", "SNMPAuthPassword": "<A-PASS>", "SNMPPrivPassword": "<P-PASS>"}

Update cray_reds_credentials.json with an encoded version of the new password.

cat <<< $(jq ".data.vault_switch_defaults=\"$(base64 --wrap=0 vault_switch_defaults.json)\"" cray_reds_credentials.json) > cray_reds_credentials.json

Verify that cray_reds_credentials.json has been updated with the new password.

jq -r '.data.vault_switch_defaults' cray_reds_credentials.json | base64 --decode

Example output:

{"SNMPUsername": "<USERID>", "SNMPAuthPassword": "<A-PASS>", "SNMPPrivPassword": "<P-PASS>"}

Replace the cray_reds_credentials secret in customizations.yaml with one containing the new credentials.

cat cray_reds_credentials.json | ./utils/secrets-encrypt.sh | yq w -f - -i customizations.yaml 'spec.kubernetes.sealed_secrets.cray_reds_credentials'

Verify that customizations.yaml contains the updated password.

./utils/secrets-decrypt.sh cray_reds_credentials | jq -r '.data.vault_switch_defaults' | base64 --decode

Example output:

{"SNMPUsername": "<USERID>", "SNMPAuthPassword": "<A-PASS>", "SNMPPrivPassword": "<P-PASS>"}