iSCSI based boot content projection solution named Scalable Boot Content Projection Service (SBPS)
is a boot content projection solution and future replacement of the Cray
Content Projection Service (CPS) and
Data Virtualization Service (DVS).
SBPS projects boot content like rootfs
and Cray Programming Environment (CPE) images.
SBPS is aimed to offer better reliability, availability, security, ease and speed of deployment and ease of management than CPS/DVS.
SBPS is introduced in CSM 1.6, where it coexists with CPS/DVS; however, SBPS is the default and CPS/DVS are deprecated.
In CSM 1.7, support is removed for projecting root filesystems and PE images using CPS and DVS.
The SBPS solution is spread across different components, including:
rootfs
and PE
) as it uses LIO (Linux IO) which is open source.Note: Using HSN for boot content projection is recommended, and use NMN for any debugging purposes. In the case that the HSN is not configured, use the NMN if it meets the bandwidth requirements.
As shown in figure #1, the basic configuration involves two iSCSI target/server (worker node) nodes
and two iSCSI initiators/clients (compute nodes or UANs) connected via HSN and/or NMN where I/O multipath
is configured. The rootfs
and PE
images are hosted in the Image Management Service (IMS)
and S3 respectively and both of these images are mapped to boot-images
bucket of S3. DNS records are created and used for target node
discovery from an initiator node during its boot.
s3fs
to mount the boot-images
bucket onto the worker nodefileio
backing storetargetcli
- LIO command-line interface to manage iSCSI devices like creation of LUNs
, listing of LUNs
, creation of fileio backstore
,
saving/clearing the configuration, and so onsystemd
service
rootfs
and PE
imagesfileio backing store
for the images to be projectedrootfs
images to be projected are tagged by BOS when the boot of initiator nodes is triggeredLUNs
for each of the fileio
backing store where the images to be projected are mapped to these LUNs
LUNs
created on the target node which has the rootfs
/ PE
images mapped are thus projected to initiator nodes when the boot is triggeredrootfs
image projected is used as part of booting the initiator node and PE
images projected are used post bootLUNs
get mounted onto the initiator node as DM multipath LUNs
Node personalization is the prerequisite step of SBPS solution where we need to first setup/configure worker nodes as iSCSI targets (servers)
with necessary provisioning, configuration and enable required components. The required RPMs for targetcli
command / LIO are part of NCN node image in CSM 1.6.
The SBPS Marshal Agent gets installed during node personalization using CFS.
This can be done in two ways:
By default worker node personalization of iSCSI SBPS is done during CSM install/upgrade
(using the Install and Upgrade Framework (IUF)).
It is initiated during bootprep (management-nodes-rollout
) in order to do worker node personalization
automatically during boot time.
Worker node personalization can be done post CSM install with CFS configuration session. Refer to Node Personalization for details.
In order to verify the readiness of the iSCSI targets before triggering the boot of compute nodes or UANs, the iSCSI configuration should be validated. See iSCSI SBPS Verification.
Once the node personalization is done and the configuration has been validated, then create BOS Session Templates with SBPS boot parameters.
There are two ways to create BOS session template:
For details, refer to Create a Session Template to Boot Compute Nodes with SBPS.
(ncn-mw#
) Obtain system name and site domain.
System name
craysys metadata get system-name
Site domain
craysys metadata get site-domain
(ncn-mw#
) Populate above values into product_vars.yaml
and then create BOS session template using sat
command.
For example:
sat bootprep run --vars-file "session_vars.yaml" --format json --bos-version v2 .bootprep-csm-1.6.0/compute-and-uan-bootprep.yaml
Refer to SAT Bootprep for further details.
Note: This way of creating BOS session template uses vcs/bootprep/compute-and-uan-bootprep.yaml
where SBPS will be chosen by default.
To initiate the boot of compute nodes or UANs, the images (rootfs
/ PE
) are tagged to determine which
rootfs
/ PE
image is to be projected.
The SBPS Marshal agent uses key/value pair of sbps-project
/true
to identify the images tagged.
rootfs
image taggingThe rootfs
images are tagged by BOS automatically when the boot of computes nodes or UANs is initiated.
Refer to
BOS Workflows for details.
It is also possible to tag the rootfs
images in IMS manually using the Cray CLI.
To tag the PE
images, first import the PE
image to IMS, and then use the Cray CLI to tag it in IMS.
Refer Import External Image to IMS for the steps to import an image to IMS.
For details on how to add or remove an IMS image tag using the Cray CLI, refer to Manage image labels.
Below are few examples.
(ncn-mw#
) Tag IMS image with ID bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb
.
cray ims images update bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb --metadata-operation set --metadata-key sbps-project --metadata-value true
(ncn-mw#
) Describe IMS image with ID bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb
.
cray ims images describe bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb --format json
Example output:
{
"arch": "x86_64",
"created": "2024-07-18T22:05:16.565885",
"id": "bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb",
"link": {
"etag": "3325f830ba9ec291005a4087be4f666f",
"path": "s3://boot-images/bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb/manifest.json",
"type": "s3"
},
"metadata": {
"sbps-project": "true" <---------------- Tagged with key/value pair sbps-project/true
},
"name": "secure-storage-ceph-6.1.94-x86_64.squashfs"
}
(ncn-mw#
) Remove tag from IMS image with ID bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb
.
cray ims images update bbe0e9eb-fa8f-4896-9f54-95dbd26de9bb --metadata-operation remove --metadata-key sbps-project
- Only remove tags from images that are not currently in use. Removing tags from images that are currently in use will stop the content projection by SBPS Marshal agent, causing undesirable behavior on compute nodes or UANs using the content.
- As mentioned in
rootfs
image tagging, BOS automatically tags therootfs
image for projection. BOS does not support automatically removing the tag, so it must be done manually.
Follow the below steps in order to boot compute nodes or UANs.
(ncn-mw#
) Use a command similar to the following to boot a single node.
cray bos sessions create --template-name <bos_session_template_name> --operation reboot --limit <xname_of_the_node>
For example, the following command creates a BOS session to boot the node with xname x3000c0s19b2n0
using the BOS session template named sbps-bos-template
.
cray bos sessions create --template-name sbps-bos-template --operation reboot --limit x3000c0s19b2n0
(ncn-mw#
) Use a command similar to the following to boot every node targeted by a session template.
cray bos sessions create --template-name <bos_session_template_name> --operation reboot
For more information on accessing the consoles of the booting nodes, see:
When booting compute nodes or UANs without the --limit
option, the boot is triggered for all the nodes targeted by the session template.
It is necessary to open the console for each node separately.
In order to monitor iSCSI SBPS target statistics, one may monitor metrics series like aggregate LUN read rate, read rate per LUN, throughput statistics on LIO portal network endpoints, and so on.
Refer to iSCSI Metrics for details.
If a user wants to continue using DVS, then during the BOS session template must use the following values in its boot sets:
rootfs_provider: "cpss3"
rootfs_provider_passthrough: "dvs:api-gw-service-nmn.local:300:hsn0,nmn0:0"
If the sat
command is used to create the BOS session template, then comment out the two lines marked SBPS and uncomment the above
two lines marked CPS in vcs/bootprep/compute-and-uan-bootprep.yaml
, and then initiate the compute node/UAN boot. Refer to the section
Content Projection Service
in the publication
HPE Cray Supercomputing User Services Software Administration Guide: CSM on HPE Cray EX Systems (S-8063)
for more details on DVS based boot content projection of rootfs
/PE
images.
Note:: The steps in Steps to achieve SBPS are not relevant and should not be followed if DVS is used.
Follow this sequence of operations:
To disable DVS, refer to the section DVS and SBPS
and to uninstall CPS, please refer to the section Uninstall CPS
under the content Scalable Boot Projection Service
documented in the publication
HPE Cray Supercomputing User Services Software Administration Guide: CSM on HPE Cray Supercomputing EX Systems (S-8063)
.