The process for upgrading Cray Systems Management (CSM) has many steps in multiple procedures which should be done in a specific order.
After the upgrade of CSM software, the CSM health checks will validate the system before doing any other operational tasks like the check and update of firmware on system components. Once the CSM upgrade has completed, other product streams for the HPE Cray EX system can be installed or upgraded.
Note: If problems are encountered during the upgrade, some of the topics do have their own troubleshooting sections, but there is also a general troubleshooting topic.
The boot order on NCNs may not be correctly set. Because of a bug, the disk entries may be listed ahead of the PXE entries. During the upgrade, a workaround is applied to fix this.
If IMS image creation CFS jobs fail, see Known Issue: IMS image creation failure for a possible workaround.
On some systems, Ceph can begin to exhibit latency over time, and if this occurs it can eventually cause services like slurm and services that are backed by etcd clusters to exhibit slowness and possible timeouts.
See Known Issue: Ceph OSD latency for a workaround.
See Prepare for Upgrade.
The upgrade of CSM software will do a controlled, rolling reboot of all management nodes before updating the CSM services.
The upgrade is a guided process starting with Upgrade Management Nodes and CSM Services.
waiting state, trying to set up volume mounts.barebones image
test may be skipped if all compute nodes are active running application workloads.(ncn-m002#) If a typescript session is already running in the shell, then first stop it with the exit command.
(ncn-m002#) Start a typescript.
script -af /root/csm_upgrade.$(date +%Y%m%d_%H%M%S).post_upgrade_health_validation.txt
export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
If additional shells are opened during this procedure, then record those with typescripts as well. When resuming a procedure after a break, always be sure that a typescript is running before proceeding.
Validate CSM health.
See Validate CSM Health.
(ncn-m002#) Stop typescripts.
For any typescripts that were started during the health validation procedure, stop them with the exit command.
(ncn-m002#) Backup upgrade logs and typescript files to a safe location.
If any typescript files are on different NCNs, then copy them to /root on ncn-m002.
Create tar file containing the logs and typescript files.
If any typescript file names are not of the form
csm_upgrade.*.txt, then append their names to the followingtarcommand in order to include them.
TARFILE="csm_upgrade.$(date +%Y%m%d_%H%M%S).logs.tgz"
tar -czvf "/root/${TARFILE}" /root/csm_upgrade.*.txt /root/output.log
Upload the tar file into S3.
This step requires that the Cray Command Line Interface is configured on the node. This should have already
been done on ncn-m002 during the upgrade process. If needed, see Configure the Cray CLI.
cray artifacts create config-data "${TARFILE}" "/root/${TARFILE}"
Review the security hardening guide, apply non-optional procedures, and review optional procedures.
See Security Hardening.
Check and update firmware if not already at the correct versions. Make sure the latest version of the HPE Cray EX HPC Firmware Pack (HFP) has been installed. Follow the procedures for updating firmware with the Firmware Actions Service (FAS) document Update Firmware with FAS.
After completion of the validation of CSM health, the CSM product stream has been fully upgraded and configured. Refer to the HPE Cray EX System Software Getting Started Guide S-8000 on the HPE Customer Support Center for more information on other product streams to be upgraded and configured after CSM.