Product Delivery

This section ensures the product content is loaded onto the system and available for later steps in the workflow.

  1. Configure optional CSM features
  2. Execute the IUF process-media and pre-install-check stages
  3. Update customizations.yaml
  4. Populate admin directory with files defining site preference
  5. Execute the IUF deliver-product stage
  6. Perform manual product delivery operations
  7. Next steps

1. Configure optional CSM features

If this IUF procedure is not part of an upgrade from CSM 1.6 to CSM 1.7, then this section should be skipped.

Rack Resiliency (experimental)

Rack Resiliency is new in CSM 1.7. It is disabled by default and it cannot change between enabled and disabled later. Administrators are advised to take time to determine whether or not they wish to use this feature. See Rack Resiliency for details.

If an administrator does not wish to enable the Rack Resiliency feature, then the rest of this section can be skipped. Otherwise, follow these steps to enable (and optionally customize) Rack Resiliency.

  1. (ncn-mw#) Retrieve the customizations.yaml file.

    TMPDIR=$(mktemp -d -p ~) &&
kubectl get secrets -n loftsman site-init -o jsonpath='{.data.customizations\.yaml}' \
    | base64 -d > "${TMPDIR}/customizations.yaml" \
    && echo "${TMPDIR}/customizations.yaml"

    Example output:

    /root/tmp.iM4FrDrJEJ/customizations.yaml

  2. (ncn-mw#) Enable the feature in customizations.yaml.

    yq write -i "${TMPDIR}/customizations.yaml" \
    'spec.kubernetes.services.rack-resiliency.enabled' "true"

  3. (ncn-mw#) Optionally, set custom zone name prefixes.

    See Zone names for details on reasons for doing this and restrictions on names. This is optional; prefixes are not required. However, prefixes cannot be changed, set, or removed later.

    1. Optionally, set a site-specific Kubernetes zone prefix.

      In the following command, replace k8s-prefix-string with the desired Kubernetes zone prefix.

      yq write -i "${TMPDIR}/customizations.yaml" \
    'spec.kubernetes.services.rack-resiliency.k8s_zone_prefix' "k8s-prefix-string"

    2. Optionally, set a site-specific Ceph zone prefix.

      In the following command, replace ceph-prefix-string with the desired Ceph zone prefix.

      yq write -i "${TMPDIR}/customizations.yaml" \
    'spec.kubernetes.services.rack-resiliency.ceph_zone_prefix' "ceph-prefix-string"

  4. (ncn-mw#) Update the site-init secret in the Kubernetes cluster.

    kubectl delete secret -n loftsman site-init \
    && kubectl create secret -n loftsman generic site-init \
        --from-file="${TMPDIR}/customizations.yaml"

    Expected output:

    secret/site-init created

  5. (ncn-mw#) Confirm that the fields are set to the desired values.

    kubectl get secrets -n loftsman site-init \
    -o jsonpath='{.data.customizations\.yaml}' \
    | base64 -d | yq r - 'spec.kubernetes.services.rack-resiliency'

    Example output (in a case where only the Ceph zone prefix was set):

    enabled: true
ceph_zone_prefix: my-ceph-prefix

2. Execute the IUF process-media and pre-install-check stages

  1. The “Install and Upgrade Framework” section of each individual product’s installation document may contain special actions that need to be performed outside of IUF for a stage. The “IUF Stage Documentation Per Product” section of the HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM (S-8052) provides a table that summarizes which product documents contain information or actions for the process-media or pre-install-check stages. Refer to that table and any corresponding product documents before continuing to the next step.

  2. Invoke iuf run with activity identifier ${ACTIVITY_NAME} and use -e to execute the process-media and pre-install-check stages. Perform the upgrade using product content found in ${MEDIA_DIR}.

    (ncn-m001#) Execute the process-media and pre-install-check stages.

    iuf -a ${ACTIVITY_NAME} -m "${MEDIA_DIR}" run -e pre-install-check

IMPORTANT* If upgrading CSM manually, ensure that the docs-csm-latest.noarch.rpm and libcsm-latest.noarch.rpm RPMs are available at path /root/<rpm> before executing the above command.

Once this step has completed:

  • Product content has been extracted from the product distribution files in ${MEDIA_DIR}
  • Pre-install checks have been performed for CSM and all products found in ${MEDIA_DIR}
  • Per-stage product hooks have executed for the process-media and pre-install-check stages

3. Update customizations.yaml

NOTE This subsection is optional and can be skipped if upgrading only CSM through IUF.

Some products require modifications to the customizations.yaml file before executing the deliver-product stage. Currently, this is limited to the Slurm and PBS Workload Manager (WLM) products, CSM Diags, and the UAN product. Refer to the “Install and Upgrade Framework” section of the Slurm, PBS, CSM Diags, and UAN product documents to determine the actions that need to be performed to update customizations.yaml.

Once this step has completed:

  • The customizations.yaml file has been updated per product documentation.

4. Populate admin directory with files defining site preferences

For creating site_vars.yaml in admin directory, refer to Populate admin directory with files defining site preferences.

5. Execute the IUF deliver-product stage

  1. The “Install and Upgrade Framework” section of each individual product’s installation document may contain special actions that need to be performed outside of IUF for a stage. The “IUF Stage Documentation Per Product” section of the HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM (S-8052) provides a table that summarizes which product documents contain information or actions for the deliver-product stage. Refer to that table and any corresponding product documents before continuing to the next step.

  2. Invoke iuf run with activity identifier ${ACTIVITY_NAME} and use -r to execute the deliver-product stage. Perform the upgrade using product content found in ${MEDIA_DIR}. Additional arguments are available to control the behavior of the deliver-product stage (for example, -rv). See the deliver-product stage documentation for details and adjust the example below if necessary.

    NOTE When installing USS 1.1 or higher, select either Slurm or PBS Pro Products to use on the system before running this stage. This should be specified in site_vars.yaml. For more information, see the deliver-product stage details in the “Install and Upgrade Framework” section of the HPE Cray Supercomputing User Services Software Administration Guide: CSM on HPE Cray Supercomputing EX Systems (S-8063).

    (ncn-m001#) Execute the deliver-product stage. Use site variables from the site_vars.yaml file found in ${ADMIN_DIR} and recipe variables from the product_vars.yaml file found in ${ADMIN_DIR}.

    iuf -a ${ACTIVITY_NAME} -m "${MEDIA_DIR}" run --site-vars \
"${ADMIN_DIR}/site_vars.yaml" -bpcd "${ADMIN_DIR}" -r deliver-product

  3. Run upload-rebuild-templates.sh to ensure the correct CSM product versions will be used by IUF now that all product artifacts have been uploaded.

    (ncn-m001#) Execute the upload-rebuild-templates.sh script.

    /usr/share/doc/csm/workflows/scripts/upload-rebuild-templates.sh

Once this step has completed:

  • Product content for all products found in ${MEDIA_DIR} has been uploaded to the system
  • Product content uploaded to the system has been recorded in the product catalog
  • Per-stage product hooks have executed for the deliver-product stage

6. Perform manual product delivery operations

NOTE This subsection is optional and can be skipped if upgrading only CSM through IUF.

NOTE This subsection is optional and can be skipped if third-party GPU and/or programming environment software is not needed.

Some products provide instructions for delivering third-party content to the system outside of IUF. If this content is desired, refer to the following documentation for instructions and execute the procedures before continuing with the workflow.

  • Content: Third-party GPU software
    • Description: User Services Software (USS) provides the gpu-nexus-tool script to upload third-party GPU software to Nexus. The GPU software is used later in the workflow when creating CFS configurations and building compute and application node images.
    • Instructions: See the “IUF Stage Details for USS” section of HPE Cray Supercomputing User Services Software Administration Guide: CSM on HPE Cray EX Systems for references to the installation procedures.
  • Content: Third-party programming environment software
    • Description: The Cray Programming Environment (CPE) provides the install-3p.sh and cpe-custom-img.sh scripts to upload third-party programming environment software to Nexus and build images. The programming environment software is used later in the workflow when creating CPE configurations.
    • Instructions: See the “CPE Install and Upgrade Framework usage” section of HPE CPE Installation Guide CSM on HPE Cray EX Systems (S-8003) for references to the installation procedures.

Once this step has completed:

  • Third-party software has been uploaded to Nexus

7. Next steps