This document guides an administrator through the patch update to Cray Systems Management v1.5.2
from CSM v1.5.0.
If upgrading from CSM v1.4.x, then follow the procedures
described in CSM major/minor version upgrade instead.
In the unusual situation of upgrading from a pre-release version of CSM
v1.5.0, then follow the procedures described in CSM major/minor version upgrade instead.
If more recent CSM v1.5 patch versions are available, note that there is no need to perform
intermediate CSM v1.5 patch upgrades. Instead, consider upgrading to the latest CSM v1.5
patch release. See CSM patch version upgrade for the full
list of patch versions.
conman in order to support Paradise hardwarerun_hms_ct_tests.sh which caused false positives for CDU switcheshmcollector loggingFASUpdate.py script also updated to accept a --timeLimit parameter to change the preset time limitOOMKilledcmsdev health checkgoss-servers being able to be installed without goss, goss-servers now requires goss as a dependencygoss-servers.service from being disabled when updating the goss-servers’s package.Validate CSM health.
See Validate CSM Health.
Run the CSM health checks to ensure that everything is working properly before the upgrade starts. After the upgrade is completed, another health check is performed. It is important to know if any problems observed at that time existed prior to the upgrade.
(ncn-m001#) Start a typescript on ncn-m001 to capture the commands and output from this procedure.
script -af csm-update.$(date +%Y-%m-%d).txt
export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
Download and extract the CSM v1.5.2 release to ncn-m001.
(ncn-m001#) Set CSM_DISTDIR to the directory of the extracted files.
IMPORTANT: If necessary, change this command to match the actual location of the extracted files.
export CSM_DISTDIR="$(pwd)/csm-1.5.2"
echo "${CSM_DISTDIR}"
(ncn-m001#) Set CSM_RELEASE_VERSION to the CSM release version.
export CSM_RELEASE_VERSION="$(${CSM_DISTDIR}/lib/version.sh --version)"
echo "${CSM_RELEASE_VERSION}"
Download and install/upgrade the latest documentation on ncn-m001.
(ncn-m001#) Run lib/setup-nexus.sh to configure Nexus and upload new CSM RPM repositories, container images, and
Helm charts:
cd "$CSM_DISTDIR"
./lib/setup-nexus.sh ; echo "RC=$?"
On success, the output should end with the following:
+ Nexus setup complete
setup-nexus.sh: OK
RC=0
In the event of an error,
consult Troubleshoot Nexus
to resolve potential problems and then try running setup-nexus.sh again. Note that subsequent runs of setup-nexus.sh
may report FAIL when uploading duplicate assets. This is okay as long as setup-nexus.sh outputs setup-nexus.sh: OK
and exits with status code 0.
(ncn-m001#) Run upgrade.sh to deploy upgraded CSM applications and services:
cd "$CSM_DISTDIR"
./upgrade.sh
On success, the output should end with the following:
+ CSM applications and services upgraded
upgrade.sh: OK
It is important to upload NCN images to IMS and to edit the cray-product-catalog. This is necessary when updating
products with IUF. If this step is skipped, IUF will fail when updating or upgrading products in the future.
(ncn-m001#) Execute script to upload CSM NCN images and update the cray-product-catalog.
/usr/share/doc/csm/upgrade/scripts/upgrade/upload-ncn-images.sh
On success, the output should end with the following:
Uploading Kubernetes images...
Uploading Ceph images...
Updating image ids...
This step updates the CFS configuration which is set as the desired configuration for the management nodes (NCNs). It ensures that the CFS configuration layers reference the correct commit hash for the version of CSM being installed. It then waits for the components to reach a configured state in CFS.
(ncn-m001#)
cd "$CSM_DISTDIR"
./update-mgmt-ncn-cfs-config.sh --base-query role=management \
--save --create-backups --clear-error
The output will look similar to the truncated output shown below.
INFO: Querying CFS configurations for the following NCNs: x3000c0s5b0n0, ...
INFO: Found configuration "management-csm-1.5.0" for component x3000c0s5b0n0
...
INFO: Updating existing layer with repo path /vcs/cray/csm-config-management.git and playbook ncn_nodes.yml
INFO: Property "commit" of layer with repo path /vcs/cray/csm-config-management.git and playbook ncn_nodes.yml updated ...
INFO: Property "name" of layer with repo path /vcs/cray/csm-config-management.git and playbook ncn_nodes.yml updated ...
INFO: No layer with repo path /vcs/cray/csm-config-management.git and playbook ncn-initrd.yml found.
INFO: Adding a layer with repo path /vcs/cray/csm-config-management.git and playbook ncn-initrd.yml to the end.
INFO: Successfully saved CFS configuration "management-csm-1.5.0-backup-20240410T205149"
INFO: Successfully saved CFS configuration "management-csm-1.5.0"
INFO: Successfully saved 1 changed CFS configuration(s) to CFS.
INFO: Updated 9 CFS components.
INFO: Waiting for 9 component(s) to finish configuration
INFO: Summary of number of components in each status: pending: 9
INFO: Waiting for 9 pending component(s)
INFO: Sleeping for 30 seconds before checking status of 9 pending component(s).
...
INFO: Sleeping for 30 seconds before checking status of 9 pending component(s).
INFO: 9 pending components transitioned to status configured: x3000c0s5b0n0, ...
INFO: Finished waiting for 9 component(s) to finish configuration.
INFO: Summary of number of components in each status: configured: 9
====> Completed update of CFS configuration(s)
====> Cleaning up install dependencies
When configuration of all components is successful, the summary line will show all components with status “configured”.
NCN images should be rebuilt at this time to acquire any changes from CSM config.
This step does not rebuild NCNs. These new images are built and stored in S3 to facilitate the add and rebuild NCN procedures.
Print the product catalog ConfigMap.
kubectl -n services get cm cray-product-catalog -o jsonpath='{.data}' | jq '. | keys'
Example outputs:
CSM running with additional products:
[
"HFP-firmware",
"analytics",
"cos",
"cos-base",
"cpe",
"cpe-aarch64",
"cray-sdu-rda",
"csm",
"csm-diags",
"hfp",
"hpc-csm-software-recipe",
"pbs",
"sat",
"sle-os-backports-15-sp3",
"sle-os-backports-15-sp4",
"sle-os-backports-sle-15-sp3-x86_64",
"sle-os-backports-sle-15-sp4-x86_64",
"sle-os-backports-sle-15-sp5-aarch64",
"sle-os-backports-sle-15-sp5-x86_64",
"sle-os-products-15-sp3",
"sle-os-products-15-sp3-x86_64",
"sle-os-products-15-sp4",
"sle-os-products-15-sp4-x86_64",
"sle-os-products-15-sp5-aarch64",
"sle-os-products-15-sp5-x86_64",
"sle-os-updates-15-sp3",
"sle-os-updates-15-sp3-x86_64",
"sle-os-updates-15-sp4",
"sle-os-updates-15-sp4-x86_64",
"sle-os-updates-15-sp5-aarch64",
"sle-os-updates-15-sp5-x86_64",
"slingshot",
"slingshot-host-software",
"slurm",
"sma",
"uan",
"uss"
]
CSM on a CSM-only system:
[
"csm"
]
Choose one of the following options based on the output from the previous step.
(ncn-m001#) Update select RPMs on the NCNs.
/usr/share/doc/csm/upgrade/scripts/upgrade/util/upgrade-test-rpms.sh
On success, the output should end with the following:
Enabling and restarting goss-servers
SUCCESS
Verify that the new CSM version is in the product catalog.
(ncn-m001#) Verify that the new CSM version is listed in the output of the following command:
kubectl get cm cray-product-catalog -n services -o jsonpath='{.data.csm}' | yq r -j - | jq -r 'to_entries[] | .key' | sort -V
Example output that includes the new CSM version (1.5.2):
0.9.2
0.9.3
0.9.4
0.9.5
0.9.6
1.0.1
1.0.10
1.2.0
1.2.1
1.2.2
1.3.0
1.3.1
1.4.0
1.4.1
1.4.2
1.4.3
1.4.4
1.5.0
1.5.1
1.5.2
Confirm that the product catalog has an accurate timestamp for the CSM upgrade.
(ncn-m001#) Confirm that the import_date reflects the timestamp of the upgrade.
kubectl get cm cray-product-catalog -n services -o jsonpath='{.data.csm}' | yq r - '"1.5.2".configuration.import_date'
(ncn-m001#) Execute the following script to take a manual backup of the Etcd clusters.
/usr/share/doc/csm/scripts/operations/etcd/take-etcd-manual-backups.sh post_patch
These clusters are automatically backed up every 24 hours, but taking a manual backup at this stage in the upgrade enables restoring from backup later in this process if needed.
This step is necessary so that nodes are using the correct images after running Update NCN images.
The rebuild will also ensure that the NCN has the latest cached container images that often accompany a CSM patch release.
Follow the Upgrade NCNs during CSM 1.5.2 Patch instructions to perform the NCN node image upgrades.
NOTE: Please follow this step if SMART disk data is needed for E1000 node.
This step is for getting the SMART data from the disks on E1000 node using the Redfish exporter into prometheus time-series database.
To configure the LDAP instance on the E1000 primary management node and reconfigure the redfish-exporter instance running on the ncn, see Configure E1000 node and Redfish Exporter.
(ncn-m001#) Remember to exit the typescript that was started at the beginning of the upgrade.
exit
It is recommended to save the typescript file for later reference.