Upgrade NCNs during CSM 1.5.2 Patch

This page provides guidance for systems that are performing an upgrade from a CSM v1.5.X release to the CSM v1.5.2 release.

The v1.5.2 upgrade page will refer to this page during NCN Upgrade.

Overview

The following steps upgrade NCNs into the node images created during the Update NCN images step of the patch procedure. The node upgrades will not change the state of the node.

It is important to do this step so that nodes are using the correct images. If NCNs are not upgraded into these images, then cloudinit will fail on the nodes the next time the nodes are rebuilt. These images were set in BSS during Update NCN images and if the NCN nodes are not upgraded, then on any node reboot will cause the node to be booted into this new image which is not following the proper upgrade procedure which could cause problems.

Steps

  1. (ncn-m001#) Upgrade storage nodes.

    1. Pick one storage node to test the first storage node upgrade on. This will be referred to as the CANARY_NODE.

      CANARY_NODE="ncn-s001"

    2. Perform the storage node upgrade on the CANARY_NODE.

      /usr/share/doc/csm/upgrade/scripts/upgrade/ncn-upgrade-worker-storage-nodes.sh ${CANARY_NODE} --upgrade

    3. After successfully upgrading the CANARY_NODE, continue upgrading the remaining storage nodes.

      1. Get a comma seperated list of storage nodes to be upgraded.

        STORAGE_NODES="$(ceph orch host ls | awk '/^ncn\-s/{if ($1 != "'"$CANARY_NODE"'") print $1}')"
STORAGE_NODES="${STORAGE_NODES//$'\n'/,}"
echo "$STORAGE_NODES"

      2. Upgrade remaining storage nodes.

        /usr/share/doc/csm/upgrade/scripts/upgrade/ncn-upgrade-worker-storage-nodes.sh "${STORAGE_NODES}" --upgrade

    For troubleshooting the storage node upgrades, see the notes in the CSM storage node upgrade procedure.

  2. (ncn-m001#) Export CSM_ARTI_DIR environment variable. (CSM_RELEASE_VERSION and CSM_DISTDIR is expected to already be set).

    CSM_REL_NAME="csm-${CSM_RELEASE_VERSION}"
export CSM_ARTI_DIR="${CSM_DISTDIR}"
echo "${CSM_ARTI_DIR}"

  3. (ncn-m001#) Update the /etc/cray/upgrade/csm/myenv file with the correct CSM_RELEASE, CSM_ARTI_DIR and CSM_REL_NAME values.

    1. Backup the previous myenv file.

      mv /etc/cray/upgrade/csm/myenv /etc/cray/upgrade/csm/myenv.old

    2. Create the new myenv file.

      cat << EOF > /etc/cray/upgrade/csm/myenv
export CSM_ARTI_DIR=${CSM_ARTI_DIR}
export CSM_RELEASE=${CSM_RELEASE}
export CSM_REL_NAME=${CSM_REL_NAME}
EOF

    3. cat the /etc/cray/upgrade/csm/myenv file to make sure it is correct.

      cat /etc/cray/upgrade/csm/myenv

  4. Upgrade master nodes and worker nodes using the following steps, these will walk through the CSM Stage 3 Upgrade Kubernetes documentation steps.

    1. (ncn-m001#) Start with step Stage 3.1 - Master node image upgrade.

    2. (ncn-m001#) Perform Stage 3.2 - Worker node image upgrade.

    3. (ncn-m002#) From ncn-m002, commence the upgrade for ncn-m001.

      1. Source the /etc/cray/upgrade/csm/myenv file to set environment variables. Also, set the CSM_RELEASE_VERSION environment variable.

        source /etc/cray/upgrade/csm/myenv
export CSM_RELEASE_VERSION="${CSM_RELEASE}"

      2. Verify that the CSM_ARTI_DIR, CSM_RELEASE, CSM_RELEASE_VERSION, and CSM_REL_NAME variables are set correctly.

        echo -e "CSM_ARTI_DIR: $CSM_ARTI_DIR\nCSM_RELEASE: $CSM_RELEASE\nCSM_RELEASE_VERSION: $CSM_RELEASE_VERSION\nCSM_REL_NAME: $CSM_REL_NAME"

      3. Perform Stage 3.3 - ncn-m001 upgrade and return to this document.

Return to CSM 1.5.2 patch

Return to the next step of the CSM 1.5.2 patch procedure Configure E1000 node and Redfish Exporter for SMART data.