management-nodes-rolloutThe management-nodes-rollout stage performs a controlled update of management NCNs by configuring them with a new CFS
configuration and/or rebuilding or upgrading them to a new image. A “rebuild” is a reboot operation that clears the
persistent OverlayFS file system on that node, i.e. all data on node-local storage will be discarded. An “upgrade” is
similar to a rebuild, but it intentionally does not clear all information off of NCN storage and master nodes. IUF will
account for the necessary minimum number of critical software instances running on the nodes to ensure
the management-nodes-rollout stage operates without impacting software availability.
NOTE management-nodes-rollout has a different procedure depending on whether or not CSM itself is being
upgraded. The two procedures differ in the handling of NCN storage nodes and NCN master nodes, but both procedures use
the same steps for rebuilding/upgrading NCN worker nodes.
If CSM is not being upgraded, then NCN storage and master nodes will not be upgraded with a new image but will be updated with a CFS configuration created in update-cfs-config.
If CSM is being upgraded, then NCN storage and master nodes will be upgraded with a new image and CFS configuration.
See
the 2. Execute the IUF management-nodes-rollout stage
documentation for more information.
management-nodes-rollout details are explained in the following sections:
ncn-m001The management-nodes-rollout stage changes the running state of the system.
The following arguments are most often used with the management-nodes-rollout stage. See iuf -h and iuf run -h for
additional arguments.
| Input | iuf Argument |
Description |
|---|---|---|
| Activity | -a ACTIVITY |
Activity created for the install or upgrade operations |
| Concurrent management rollout percentage | -cmrp CONCURRENT_MANAGEMENT_ROLLOUT_PERCENTAGE |
Percentage value that limits the number of NCN worker nodes rolled out in parallel |
| Limit management rollout list | --limit-management-rollout LIMIT_MANAGEMENT_ROLLOUT |
List of NCN management nodes to be rolled out, specified by HSM role and subrole (Management_Master, Management_Worker, Management_Storage) or by NCN hostname (e.g. ncn-w003, ncn-s001, etc.). For each node supplied in the list, ensure that the label iuf-prevent-rollout=true is not set. If it is set, remove the label using the command: kubectl label nodes "${NODE}" --overwrite iuf-prevent-rollout- |
The code executed by this stage exists within IUF. See the management-nodes-rollout entry
in /usr/share/doc/csm/workflows/iuf/stages.yaml and the corresponding files
in /usr/share/doc/csm/workflows/iuf/operations/
for details on the commands executed.
There are two methods administrators can use to avoid rollouts on specific NCN management nodes: by specifying
the --limit-management-rollout argument for specifying which group(s) of nodes should be rebuilt or by using kubectl
to label nodes with iuf-prevent-rollout=true.
When NCN worker nodes are being rebuilt/upgraded and when NCN storage nodes are being upgraded, an additional Argo
workflow will execute in addition to the standard IUF Argo workflows. It will include the string ncn-lifecycle-rebuild
in its name.
The -cmrp argument limits the percentage of worker nodes rolled out in parallel. The worker node rebuild can
coordinate rebuilding multiple worker nodes at once. It starts by rebuilding one worker node. Once that node has been
removed from the system, the workflow checks if it is safe to rebuild the next worker node based on what services are
running in the system. If it is safe, it will proceed to rebuild the next node, partially in parallel with the first
worker node rebuild. If it is unsafe to rebuild in parallel because the system could get into a bad state, then it waits
to rebuild the second node until it is safe. The -cmrp parameter selects the percentage of worker nodes that the
worker node rebuild should coordinate rebuilding at one time. For example, if there are 15 worker nodes and -cmrp 33
is specified, then 5 worker nodes will be rebuilt at once and with as much parallelization as possible given the state
of the system. Note that the system admin’s discretion should be used when deciding the value of -cmrp.
The largest number of management worker nodes that has been tested rebuilding in parallel is 5 nodes.
Management master and storage nodes only need to upgraded when performing a CSM upgrade. If not performing a CSM upgrade, then NCN master nodes should not be upgraded with a new image and should only be configured with the new CFS configuration created during the update-cfs-config stage.
NOTE This section describes how to manually rebuild/upgrade a worker node outside of IUF with an image and CFS configuration created through IUF. This is not the normal procedure that IUF uses for rebuilding/upgrading NCN worker nodes. This procedure should be followed if NCN worker nodes need to be rebuilt or upgraded outside of IUF.
The upgrade and rebuild procedures for NCN worker nodes are identical. These instructions apply to both NCN worker node upgrades and NCN worker node rebuilds. The words ‘rebuild’ and ‘upgrade’ are exchangeable in this section.
Get the image ID and CFS configuration created for worker nodes during the prepare-images and update-cfs-config
stages. Follow the instructions in the prepare-images Artifacts created
documentation to get the values for final_image_id and configuration for images with a configuration_group_name
value matching Management_Worker. These values will be used in the next step.
Upgrade/rebuild the worker node. The worker node is automatically rebuilt using Argo workflows. If rebuilding multiple worker nodes at once, see this page for restrictions.
(ncn-m001#) Rebuild a worker node. Use the values acquired in the previous step in place of <final_image_id>
and <configuration>.
/usr/share/doc/csm/upgrade/scripts/upgrade/ncn-upgrade-worker-storage-nodes.sh ncn-w001 --image-id <final_image_id> --desired-cfs-conf <configuration>
In general, worker node rebuilds should complete successfully before starting another rebuild. A worker node can get into a bad state if it has been partially rebuilt and an attempt is made to restart the rebuild on that same node. As a result, it is not possible to start another worker node rebuild if there is an existing incomplete worker node rebuild workflow, where “incomplete” means it has stopped before successfully completing the full workflow. If an incomplete workflow exists and an attempt is made to start another worker rebuild workflow, the original incomplete worker rebuild workflow will continue and no new workflow will be created.
If it is necessary to start an entirely new worker rebuild workflow after a previous worker rebuild workflow failed, the failed workflow must be deleted from Kubernetes first.
WARNING Deleting a workflow will delete information about the state of that workflow and the steps that have been
completed. Deleting a partially complete workflow should be done cautiously and only if needed.
To delete a failed Argo workflow, complete the following steps.
Get the name of the failed workflow. All worker rebuild workflows start with ncn-lifecycle-rebuild. The name of the
worker rebuild workflow can be found in the Argo UI or by searching workflows in Kubernetes with the following
command.
(ncn-m#) List failed worker rebuild workflows. Note that NCN storage upgrade workflows will also
contain ncn-lifecycle-rebuild in their name and may be present in this list.
kubectl get workflows -n argo | grep 'ncn-lifecycle-rebuild' | grep 'Fail'
(ncn-m#) Delete the failed workflow.
kubectl delete workflows -n argo <failed workflow>
After deleting the failed workflow, a new worker rebuild workflow can be started.
(ncn-m001#) Execute the management-nodes-rollout stage for activity admin-230127 using the default concurrent
management rollout percentage and limiting the operation to Management_Worker nodes.
iuf -a admin-230127 run --limit-management-rollout Management_Worker -r management-nodes-rollout
Expected behavior: All NCN worker nodes will be rebuilt. Each set of worker nodes that is being rebuilt will contain 20% of the total number of worker nodes. For example, if there are 10 total worker nodes, then 2 will be rebuilt at a time.
(ncn-m001#) Execute the management-nodes-rollout stage for activity admin-230127 using the default concurrent
management rollout percentage and limiting the operation to ncn-s001 ncn-s002 nodes.
iuf -a admin-230127 run --limit-management-rollout ncn-s001 ncn-s002 -r management-nodes-rollout
Expected behavior:
ncn-s001 will be upgradedncn-s002 will be upgraded(ncn-m001#) Execute the management-nodes-rollout stage for activity admin-230127 using the following parameters.
ncn-w001 through ncn-w010)--limit-management-rollout Management_Worker-cmrp 33ncn-w004 is labeled with iuf-prevent-rollout=trueFirst, label ncn-w004 with iuf-prevent-rollout=true. Then execute the following command.
iuf -a admin-230127 run --limit-management-rollout Management_Worker --cmrp 33 -r management-nodes-rollout
Expected behavior:
ncn-w001,ncn-w002,ncn-w003 will be upgradedncn-w005,ncn-w006,ncn-w007 will be upgradedncn-w008,ncn-w009,ncn-w010 will be upgradedncn-m001Follow these steps when upgrading ncn-m001
during 2.1 management-nodes-rollout with CSM upgrade
when following the procedures in
Install or upgrade additional products with IUF
or Upgrade CSM and additional products with IUF.
(ncn-mw#) Set the IMS_RESULTANT_IMAGE_ID to be the final_image_id found
in 2.1 management-nodes-rollout with CSM upgrade. Then update the BSS parameters for ncn-m001.
IMS_RESULTANT_IMAGE_ID=<value of final_image_id>
XNAME=$(ssh ncn-m001 cat /etc/cray/xname)
echo $XNAME
/usr/share/doc/csm/scripts/operations/node_management/assign-ncn-images.sh \
-p $IMS_RESULTANT_IMAGE_ID $XNAME