Managed Rollout

This section updates the software running on managed compute and application (UAN, etc.) nodes.

  1. Update managed host firmware (FAS)
  2. Execute the IUF managed-nodes-rollout stage
    1. LNet router nodes and gateway nodes
    2. Compute nodes
      1. Preparation
      2. Reboot compute nodes
      3. Compute reboot complete
    3. Application nodes
  3. Update managed host Slingshot NIC firmware
  4. Execute the IUF post-install-check stage
  5. Next steps

1. Update managed host firmware (FAS)

Refer to Update Firmware with FAS for details on how to upgrade the firmware on managed nodes using FAS.

Once this step has completed:

  • Host firmware has been updated on managed nodes.

2. Execute the IUF managed-nodes-rollout stage

This section describes how to update software on managed nodes. It describes how to test a new IMS image and CFS configuration on a single “canary node” first before rolling it out to the other managed nodes. Modify the procedure as necessary to accommodate site preferences for rebooting managed nodes. If the system has heterogeneous nodes, it may be desirable to repeat this process with multiple canary nodes, one for each distinct node configuration. The IMS images, CFS configurations, and BOS session templates used are created by the prepare-images stage; see the prepare-images Artifacts created documentation for details on how to query the IMS images and CFS configurations.

NOTE Additional arguments are available to control the behavior of the managed-nodes-rollout stage. See the managed-nodes-rollout stage documentation for details and adjust the examples below if necessary.

2.1 LNet router nodes and gateway nodes

LNet router nodes or gateway nodes should be upgraded before rebooting compute nodes to new IMS images and CFS configurations. Because LNet routers and gateway nodes are examples of application nodes, the instructions in this section are the same as in 2.3 Application nodes.

Because LNet router nodes and gateway nodes are not managed by workload managers, the IUF managed-nodes-rollout stage cannot reboot them in a controlled manner via the -mrs stage argument. The IUF managed-nodes-rollout stage can reboot LNet router and gateway nodes using the -mrs reboot argument; however, this is not recommended, because an immediate reboot of the nodes is likely to be disruptive to users and overall system health. Administrators should determine the best approach for rebooting LNet router and gateway nodes outside of IUF that aligns with site preferences.

Once this step has completed:

  • Managed LNet router and gateway nodes (if any) have been rebooted to the IMS images and CFS configurations created in previous steps of this workflow.
  • Per-stage product hooks have executed for the managed-nodes-rollout stage if IUF managed-nodes-rollout procedures were used to perform the reboots.

2.2 Compute nodes

2.2.1 Preparation

  1. The “Install and Upgrade Framework” section of each individual product’s installation document may contain special actions that need to be performed outside of IUF for a stage. The “IUF Stage Documentation Per Product” section of the HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM (S-8052) provides a table that summarizes which product documents contain information or actions for the managed-nodes-rollout stage. Refer to that table and any corresponding product documents before continuing to the next step.

  2. Before booting computes, consider if SMA OpenSearch needs to be tuned. Refer to the “Configure OpenSearch” section in the HPE Cray EX System Monitoring Application Administration Guide for instructions on tuning OpenSearch.

  3. Invoke iuf run with -r to execute the managed-nodes-rollout stage on a single node to ensure the node reboots successfully with the desired IMS image and CFS configuration. This node is referred to as the “canary node” in the remainder of this section. Use --limit-managed-rollout to target the canary node only and use -mrs reboot to reboot the canary node immediately.

    (ncn-m001#) Execute the managed-nodes-rollout stage with a single xname, rebooting the canary node immediately. Replace the example value of ${XNAME} with the xname of the canary node.

    XNAME=x3000c0s29b1n0
iuf -a "${ACTIVITY_NAME}" -m "${MEDIA_DIR}" run -r managed-nodes-rollout --limit-managed-rollout "${XNAME}" -mrs reboot

  4. Verify that the canary node booted successfully with the desired IMS image and CFS configuration.

2.2.2 Reboot compute nodes

If PBS is the workload manager, then proceed to Reboot compute nodes with PBS. If Slurm is the workload manager, then proceed to Reboot compute nodes with Slurm.

Reboot compute nodes with PBS

  1. Create a maintenance reservation in PBS. For more information on maintenance reservations, see the PBS Professional Administrator’s Guide.

  2. (ncn-m001#) After the reservation has started, execute the managed-nodes-rollout stage with the -mrs reboot option to immediately reboot all compute nodes.

    iuf -a "${ACTIVITY_NAME}" -m "${MEDIA_DIR}" run -r managed-nodes-rollout -mrs reboot

  3. Proceed to 2.2.3 Compute reboot complete.

Reboot compute nodes with Slurm

  1. Configure Slurm to stage data in BOS.

    Follow the instructions in the section Using staged sessions with Slurm of the Rolling Upgrades using BOS documentation. These instructions describe two parameters that must be set in the slurm.conf file. Return to these instructions after setting them.

  2. (ncn-m001#) Execute the managed-nodes-rollout stage.

    This will not reboot the nodes immediately; instead, it will stage data to BOS and allow the workload manager to reboot the nodes when it is ready to do so.

    If an immediate reboot of compute nodes is desired instead, add -mrs reboot to the iuf run command. NOTE: If the -mrs reboot option is used with Slurm, then skip the remaining steps in this section and proceed directly to 2.2.3 Compute reboot complete.

    iuf -a "${ACTIVITY_NAME}" -m "${MEDIA_DIR}" run -r managed-nodes-rollout

  3. List all of the compute nodes in the system by their node IDs (NIDs).

    NOTE: If the -mrs reboot option was used with Slurm in the previous step, then skip the remaining steps in this section and proceed directly to 2.2.3 Compute reboot complete.

    1. (ncn-m001#) Enter the SAT container.

      sat bash

    2. (sat-container#) Fetch the compute list:

      sat status --fields xname --filter role=compute --no-headings --no-borders | xargs sat xname2nid

      Example output:

      nid[000001-000004]

    3. (sat-container#) Exit the SAT container.

      exit

  4. Tell Slurm to reboot the compute nodes.

    This only works for compute nodes, and they must be specified explicitly. Use the compute list output from the previous step as the last argument.

    (compute#) A sample reboot command to reboot NIDs 1 through 4.

    scontrol reboot nextstate=Resume Reason="IUF Managed Nodes Rollout" nid[000001-000004]

  5. Proceed to 2.2.3 Compute reboot complete.

2.2.3 Compute reboot complete

Once this step has completed:

  • Managed compute nodes have been rebooted to the IMS images and CFS configurations created in previous steps of this workflow.
  • Per-stage product hooks have executed for the managed-nodes-rollout stage.

2.3 Application nodes

NOTE If LNet router or gateway nodes were upgraded in the 2.1 LNet router nodes and gateway nodes section, there is no need to upgrade them again in this section. Follow the instructions in this section to upgrade any remaining application nodes (UANs, etc.) that have not been upgraded yet.

Because application nodes are not managed by workload managers, the IUF managed-nodes-rollout stage cannot reboot them in a controlled manner via the -mrs stage argument. The IUF managed-nodes-rollout stage can reboot application nodes using the -mrs reboot argument; however, this is not recommended, because an immediate reboot of the nodes is likely to be disruptive to users and overall system health. Administrators should determine the best approach for rebooting application nodes outside of IUF that aligns with site preferences.

Once this step has completed:

  • Managed application nodes (UAN, etc.) have been rebooted to the IMS images and CFS configurations created in previous steps of this workflow.
  • Per-stage product hooks have executed for the managed-nodes-rollout stage if IUF managed-nodes-rollout procedures were used to perform the reboots.

3. Update managed host Slingshot NIC firmware

If new Slingshot NIC firmware was provided, refer to the “200Gbps NIC Firmware Management” section of the HPE Slingshot Installation Guide for Bare Metal for details on how to update NIC firmware on managed nodes.

Reboot the managed nodes for which Slingshot NIC firmware has been updated. Administrators should determine the best approach for rebooting managed nodes that aligns with site preferences.

Once this step has completed:

  • Slingshot NIC firmware has been updated on managed nodes.

4. Execute the IUF post-install-check stage

  1. The “Install and Upgrade Framework” section of each individual product’s installation document may contain special actions that need to be performed outside of IUF for a stage. The “IUF Stage Documentation Per Product” section of the HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM (S-8052) provides a table that summarizes which product documents contain information or actions for the post-install-check stage. Refer to that table and any corresponding product documents before continuing to the next step.

  2. Invoke iuf run with -r to execute the post-install-check stage.

    (ncn-m001#) Execute the post-install-check stage.

    iuf -a "${ACTIVITY_NAME}" -m "${MEDIA_DIR}" run -r post-install-check

Once this step has completed:

  • Per-stage product hooks have executed for the post-install-check stage to verify product software is executing as expected.

5. Next steps