Update Default Air-Cooled BMC and Leaf-BMC Switch SNMP Credentials

This procedure updates the default credentials used when new air-cooled hardware is discovered for the first time. This includes the default Redfish credentials used for new air-cooled NodeBMCs and Slingshot switch BMCs (RouterBMCs), and SNMP credentials for new management leaf-BMC switches.

IMPORTANT After this procedure is completed, all future air-cooled hardware added to the system will be assumed to be configured with the new global default credential.

NOTE This procedure will not update the Redfish or SNMP credentials for existing air-cooled devices. To change the credentials on existing air-cooled hardware follow the Change Air-Cooled Node BMC Credentials and Configuring SNMP in CSM procedures.

Limitation

The default global credentials used for liquid-cooled BMCs in the Change Cray EX Liquid-Cooled Cabinet Global Default Password procedure needs to be the same as the one used in this procedure for air-cooled BMCs (River hardware).

Procedure

The River Endpoint Discovery Service (REDS) sealed secret contains the default global credential used by REDS.

1. Update the default credentials used by REDS

Follow the Redeploying a Chart procedure with the following specifications:

  • Chart name: cray-hms-reds

  • Base manifest name: core-services

  • (ncn-mw#) When reaching the step to update the customizations, perform the following steps:

    Only follow these steps as part of the previously linked chart redeploy procedure.

    1. Run git clone https://github.com/Cray-HPE/csm.git.

    2. Acquire sealed secret keys.

      mkdir -pv certs
      kubectl -n kube-system get secret sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d > certs/sealed_secrets.crt
      kubectl -n kube-system get secret sealed-secrets-key -o jsonpath='{.data.tls\.key}' | base64 -d > certs/sealed_secrets.key
      
    3. Modify REDS sealed secret to use new global default credentials.

      1. Inspect the original default Redfish credentials used by REDS and HMS discovery.

        ./utils/secrets-decrypt.sh cray_reds_credentials ./certs/sealed_secrets.key ./customizations.yaml | jq .data.vault_redfish_defaults -r | base64 -d | jq
        

        Expected output looks similar to the following:

        {
            "Cray": {
                "Username": "root",
                "Password": "foo"
            }
        }
        
      2. Inspect the original default switch SNMP credentials used by REDS and HMS discovery.

        ./utils/secrets-decrypt.sh cray_reds_credentials ./certs/sealed_secrets.key ./customizations.yaml | jq .data.vault_switch_defaults -r | base64 -d | jq
        

        Expected output looks similar to the following:

        {
            "SNMPUsername": "testuser",
            "SNMPAuthPassword": "foo",
            "SNMPPrivPassword": "bar"
        }
        
      3. Update the default credentials in customizations.yaml for REDS and HMS discovery to use.

        1. Specify the desired default Redfish credentials.

          echo '{"Cray":{"Username":"root","Password":"foobar"}}' | base64 > reds.redfish.creds.json.b64
          
        2. Specify the desired default SNMP credentials.

          echo '{"SNMPUsername":"testuser","SNMPAuthPassword":"foo1","SNMPPrivPassword":"bar2"}' | base64 > reds.switch.creds.json.b64
          
      4. Update and regenerate the cray_reds_credentials sealed secret.

        cat << EOF | yq w - 'data.vault_redfish_defaults' "$(<reds.redfish.creds.json.b64)" | yq w - 'data.vault_switch_defaults' "$(<reds.switch.creds.json.b64)" | yq r -j - | ./utils/secrets-encrypt.sh | yq w -f - -i ./customizations.yaml 'spec.kubernetes.sealed_secrets.cray_reds_credentials'
        {
            "kind": "Secret",
            "apiVersion": "v1",
            "metadata": {
                "name": "cray-reds-credentials",
                "namespace": "services",
                "creationTimestamp": null
            },
            "data": {}
        }
        EOF
        
      5. Decrypt generated secret for review.

        1. Review the default Redfish credentials.

          ./utils/secrets-decrypt.sh cray_reds_credentials ./certs/sealed_secrets.key ./customizations.yaml | jq .data.vault_redfish_defaults -r | base64 -d | jq
          

          Expected output looks similar to the following:

          {
              "Username": "root",
              "Password": "foobar"
          }
          
        2. Review the default switch SNMP credentials.

          ./utils/secrets-decrypt.sh cray_reds_credentials ./certs/sealed_secrets.key ./customizations.yaml | jq .data.vault_switch_defaults -r | base64 -d | jq
          

          Expected output looks similar to the following:

          {
              "SNMPUsername": "testuser",
              "SNMPAuthPassword": "foo1",
              "SNMPPrivPassword": "bar2"
          }
          
  • (ncn-mw#) When reaching the step to validate the redeployed chart, perform the following steps:

    Only follow these steps as part of the previously linked chart redeploy procedure.

    1. Wait for the REDS Vault loader job to run to completion.

      kubectl -n services wait job cray-reds-vault-loader --for=condition=complete --timeout=5m
      
    2. Verify that the default Redfish credentials have updated in Vault.

      VAULT_PASSWD=$(kubectl -n vault get secrets cray-vault-unseal-keys -o json | jq -r '.data["vault-root"]' |  base64 -d)
      kubectl -n vault exec -it cray-vault-0 -c vault -- env VAULT_TOKEN=$VAULT_PASSWD VAULT_ADDR=http://127.0.0.1:8200 vault kv get secret/reds-creds/defaults
      

      Expected output:

      ==== Data ====
      Key     Value
      ---     -----
      Cray    map[password:foobar username:root]
      
    3. Verify that the default SNMP credentials have updated in Vault.

      kubectl -n vault exec -it cray-vault-0 -c vault -- env VAULT_TOKEN=$VAULT_PASSWD VAULT_ADDR=http://127.0.0.1:8200 vault kv get secret/reds-creds/switch_defaults
      

      Expected output:

      ========== Data ==========
      Key                 Value
      ---                 -----
      SNMPAuthPassword    foo1
      SNMPPrivPassword    bar2
      SNMPUsername        testuser
      
  • Make sure to perform the entire linked procedure, including the step to save the updated customizations.

2. Restart the SNMP-backed RTS to pick up the SNMP credential changes

  1. Scale the SNMP-backed RTS down.

    kubectl scale deployment cray-hms-rts-snmp -n services --replicas=0
    
  2. Scale the SNMP-backed RTS up.

    kubectl scale deployment cray-hms-rts-snmp -n services --replicas=1