SAT provides an automated solution for creating CFS configurations, building
and configuring images in IMS, and creating BOS session templates. The
solution is based on a given input file that defines how those configurations,
images, and session templates should be created. This automated process centers
around the sat bootprep command. man page documentation for sat bootprep
can be viewed similar to other SAT commands. For more information, see
SAT Man Pages.
(ncn-m001#) For example:
sat-man sat-bootprep
The sat bootprep command helps the Install and Upgrade Framework (IUF)
install, upgrade, and deploy products on systems managed by CSM. Outside of IUF,
it is uncommon to use sat bootprep. For more information on this relationship,
see SAT and IUF. For more information on IUF, see
Install and Upgrade Framework.
bootprep vs SAT bootsyssat bootprep is used to create CFS configurations, build and
rename IMS images, and create BOS session templates which tie the
configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
including (but not limited to) performing BOS operations (such as creating BOS
sessions), powering on and off cabinets, and checking the state of the system
prior to shutdown.
bootprep input fileThe input file provided to sat bootprep is a YAML file containing information
which CFS, IMS, and BOS use to create configurations, images, and BOS session
templates respectively. Writing and modifying these input files is the main task
associated with using sat bootprep. An input file is composed of three main
sections, one each for configurations, images, and session templates. These
sections may be specified in any order, and any of the sections may be omitted
if desired.
The sat bootprep input file is validated against a versioned schema
definition. The input file should specify the version of the schema with which
it is compatible under a schema_version key. For example:
---
schema_version: 1.0.2
(ncn-m001#) The current sat bootprep input file schema version can be viewed with the
following command:
sat bootprep view-schema | grep '^version:'
Example output:
version: '1.0.2'
The sat bootprep run command validates the schema version specified
in the input file. The command also makes sure that the schema version
of the input file is compatible with the schema version understood by the
current version of sat bootprep. For more information on schema version
validation, see the schema_version property description in the bootprep
input file schema. For more information on viewing the bootprep input file
schema in either raw form or user-friendly HTML form, see View SAT Bootprep
Schema.
The default HPC CSM Software Recipe bootprep input files provided by the
hpc-csm-software-recipe release distribution already contain the correct
schema version.
The CFS configurations are defined under a configurations key. Under this
key, list one or more configurations to create. For each
configuration, give a name in addition to the list of layers that
comprise the configuration.
Each layer can be defined by a product name and optionally a version number,
commit hash, or branch in the product’s configuration repository. If this
method is used, the layer is created in CFS by looking up relevant configuration
information (including the configuration repository and commit information) from
the cray-product-catalog Kubernetes ConfigMap as necessary. A version may be
supplied. However, if it is absent, the version is assumed to be the latest
version found in the cray-product-catalog.
Alternatively, a configuration layer can be defined by explicitly referencing
the desired configuration repository. Specify the intended version
of the Ansible playbooks by providing a branch name or commit hash with branch
or commit.
The following example shows a CFS configuration with two layers. The first layer is defined in terms of a product name and version, and the second layer is defined in terms of a Git clone URL and branch:
---
configurations:
- name: example-configuration
layers:
- name: example-product
playbook: example.yml
product:
name: example
version: 1.2.3
- name: another-example-product
playbook: another-example.yml
git:
url: "https://vcs.local/vcs/another-example-config-management.git"
branch: main
A CFS configuration may also contain the following keys:
if_exists key to specify the behaviour when an image already exists on
the system with the same name, as described in Set behavior when items already
exist.When sat bootprep is run against an input file, a CFS configuration is created
corresponding to each configuration in the configurations section. For
example, the configuration created from an input file with the layers listed
above might look something like the following:
{
"lastUpdated": "2022-02-07T21:47:49Z",
"layers": [
{
"cloneUrl": "https://vcs.local/vcs/example-config-management.git",
"commit": "<commit hash>",
"name": "example product",
"playbook": "example.yml"
},
{
"cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
"commit": "<commit hash>",
"name": "another example product",
"playbook": "another-example.yml"
}
],
"name": "example-configuration"
}
The IMS images are defined under an images key. Under the images key, the
user may define one or more images to be created in a list. Each element of the
list defines a separate IMS image to be built and/or configured. Images must
contain a name key and a base key.
The name key defines the name of the resulting IMS image. The base key
defines the base image to be configured or the base recipe to be built and
optionally configured. One of the following keys must be present under the
base key:
ims key to specify an existing image or recipe in IMS.product key to specify an image or recipe provided by a particular
version of a product. If a product provides more than one image or recipe,
specify a filter to select one. For more information, see
Filter Base Images or Recipes from a Product.image_ref key to specify another image from the input file
using its ref_name.Images may also contain the following keys:
configuration key to specify a CFS configuration with which to
customize the built image. If a configuration is specified, then configuration
groups must also be specified using the configuration_group_names key.ref_name key to specify a unique name that can refer to this image
within the input file in other images or in session templates. The ref_name
key allows references to images from the input file that have dynamically
generated names as described in
Dynamic Variable Substitutions.description key to describe the image in the bootprep input file.
Note that this key is not currently used.if_exists key to specify the behaviour when an image already exists on
the system with the same name, as described in Set behavior when items already
exist.Here is an example of an image using an existing IMS recipe as its base. This
example builds an IMS image from that recipe. It then configures it with
a CFS configuration named example-compute-config. The example-compute-config
CFS configuration can be defined under the configurations key in the same
input file, or it can be an existing CFS configuration. Running sat bootprep
against this input file results in an image named example-compute-image.
images:
- name: example-compute-image
description: >
An example compute node image built from an existing IMS recipe.
base:
ims:
name: example-compute-image-recipe
type: recipe
configuration: example-compute-config
configuration_group_names:
- Compute
Here is an example showing the definition of two images. The first image is
built from a recipe provided by the uss product. The second image uses the
first image as a base and configures it with a configuration named
example-compute-config. The value of the first image’s ref_name key is used
in the second image’s base.image_ref key to specify it as a dependency.
Running sat bootprep against this input file results in two images, the
first named example-uss-image and the second named example-compute-image.
images:
- name: example-uss-image
ref_name: example-uss-image
description: >
An example image built from the recipe provided by the USS product.
base:
product:
name: uss
version: 1.0.0
type: recipe
- name: example-compute-image
description: >
An example image that is configured from an image built from the recipe provided
by the USS product.
base:
image_ref: example-uss-image
configuration: example-compute-config
configuration_group_names:
- Compute
This example assumes that the given version of the uss product provides
only a single IMS recipe. If more than one recipe is provided by the
given version of the uss product, use a filter as described in
Filter Base Images or Recipes from a Product.
A product may provide more than one image or recipe. If this happens,
filter the product’s images or recipes whenever a base image or recipe from
that product is used. Beneath the base.product value within an image,
specify a filter key to create a filter using the following criteria:
prefix key to filter based on a prefix matching the name of the
image or recipe.wildcard key to filter based on a shell-style wildcard matching the
name of the image or recipe.arch key to filter based on the target architecture of the image or
recipe in IMS.When specifying more than one filter key, all filters must match only the desired image or recipe. An error occurs if either no images or recipes match the given filters or if more than one image or recipe matches the given filters.
Here is an example of three IMS images built from the Kubernetes image and the
Ceph storage image provided by the csm product. This example uses a prefix
filter to select from the multiple images provided by the CSM product.
The first two IMS images in the example find any image from the specified csm
product version whose name starts with secure-kubernetes. The third image in
the example finds any csm image whose name starts with secure-storage-ceph.
All three images are then configured with a configuration named
example-management-config. Running sat bootprep against this input file
results in three IMS images named worker-example-csm-image,
master-example-csm-image, and storage-example-csm-image.
images:
- name: worker-example-csm-image
base:
product:
name: csm
version: 1.4.1
type: image
filter:
prefix: secure-kubernetes
configuration: example-management-config
configuration_group_names:
- Management_Worker
- name: master-example-csm-image
base:
product:
name: csm
version: 1.4.1
type: image
filter:
prefix: secure-kubernetes
configuration: example-management-config
configuration_group_names:
- Management_Master
- name: storage-example-csm-image
base:
product:
name: csm
version: 1.4.1
type: image
filter:
prefix: secure-storage-ceph
configuration: example-management-config
configuration_group_names:
- Management_Storage
Here is an example of two IMS images built from recipes provided by the uss
product. This example uses an architecture filter to select from the multiple
recipes provided by the USS product. The first image will be built from the
x86_64 version of the IMS recipe provided by the specified version of the
uss product. The second image will be built from the aarch64 version of
the IMS recipe provided by the specified version of the uss product.
images:
- name: example-uss-image-x86_64
ref_name: example-uss-image-x86_64
description: >
An example image built from the x86_64 recipe provided by the USS product.
base:
product:
name: uss
version: 1.0.0
type: recipe
filter:
arch: x86_64
- name: example-uss-image-aarch64
ref_name: example-uss-image-aarch64
description: >
An example image built from the aarch64 recipe provided by the USS product.
base:
product:
name: uss
version: 1.0.0
type: recipe
filter:
arch: aarch64
The BOS session templates are defined under the session_templates key. Each
session template must provide values for the name, image, configuration,
and bos_parameters keys. The name key defines the name of the resulting BOS
session template. The image key defines the image to use in the BOS session
template. One of the following keys must be present under the image key:
ims key to specify an existing image or recipe in IMS.image_ref key to specify another image from the input file
using its ref_name.The configuration key defines the CFS configuration specified
in the BOS session template.
The bos_parameters key defines parameters that are passed through directly to
the BOS session template. The bos_parameters key should contain a boot_sets
key, and each boot set in the session template should be specified under
boot_sets. Each boot set can contain the following keys, all of
which are optional:
arch key to specify the architecture of the nodes that should be
targeted by the boot set. Valid values are the same as those used by
Hardware State Manager (HSM).kernel_parameters key to specify the parameters passed to the kernel
on the command line.network key to specify the network over which the nodes boot.node_list key to specify the nodes to add to the boot set.node_roles_groups key to specify the HSM roles to add to the boot
set.node_groups key to specify the HSM groups to add to the boot set.rootfs_provider key to specify the root file system provider.rootfs_provider_passthrough key to specify the parameters to add to
the rootfs= kernel parameter.if_exists key to specify the behaviour when an image already exists on
the system with the same name, as described in Set behavior when items already
exist.As mentioned above, the parameters under bos_parameters are passed through
directly to BOS. For more information on the properties of a BOS boot set,
see BOS Session Templates.
Here is an example of a BOS session template that refers to an existing IMS
image by name and targets nodes with the role Compute and the architecture
X86 in HSM. This session template uses SBPS as the rootfs_provider:
session_templates:
- name: example-session-template
image:
ims:
name: example-image
configuration: example-configuration
bos_parameters:
boot_sets:
example_boot_set:
arch: X86
kernel_parameters: ip=dhcp quiet
node_roles_groups:
- Compute
rootfs_provider: sbps
rootfs_provider_passthrough: sbps:v1:iqn.2023-06.csm.iscsi:_sbps-hsn._tcp.{{default.system-name}}.{{default.site-domain}}:300
(NOTE: Make sure to define the values for system-nameand site-domain in the site-vars.yaml file.)
Here is an example of a BOS session template that refers to an image from the
input file by its ref_name and targets nodes with the role Compute and the
architecture ARM in HSM. Note that using the image_ref key requires that
an image defined in the input file specifies example-image as the value of
its ref_name key.
session_templates:
- name: example-session-template
image:
image_ref: example-image
configuration: example-configuration
bos_parameters:
boot_sets:
example_boot_set:
arch: ARM
kernel_parameters: ip=dhcp quiet
node_roles_groups:
- Compute
rootfs_provider: sbps
rootfs_provider_passthrough: sbps:v1:iqn.2023-06.csm.iscsi:_sbps-hsn._tcp.{{default.system-name}}.{{default.site-domain}}:300
(NOTE: Make sure to define the values for system-nameand site-domain in the site-vars.yaml file.)
The sat bootprep command supports variable substitution in certain fields of
the input file using Jinja2 template syntax. Variable definitions are sourced
from the following locations in order from highest to lowest precedence:
--vars option--vars-file option--recipe-version option.product_vars.yaml in the
hpc-csm-software-recipe repository in VCS. This file contains variables
associated with each product provided in the recipe.product_vars.yaml file delivered with the HPC CSM Software
Recipe now contains more complex variables that can only be resolved by
the Install and Upgrade Framework (IUF). See SAT and IUF
for more information.Beginning with the introduction of IUF in CSM 1.4.0, the default sat bootprep input files now use more complex variables that are only resolved
properly by IUF. It is still possible to call sat bootprep directly with
variables defined in a file, but if the default bootprep input files are used,
the session_vars.yaml file from an IUF activity must be passed to sat bootprep. See SAT and IUF for details.
This section describes how to view the variables defined by a given installed
version of the HPC CSM Software Recipe. As described in Variable Substitutions,
these variables must be rendered by IUF before they can be used by sat bootprep.
(ncn-m001) View a listing of the default HPC CSM Software Recipe variables and
their values by running sat bootprep list-vars.
sat bootprep list-vars
(ncn-m001) The variables for a specific version of the HPC CSM Software Recipe
can also be listed using the --recipe-version option.
sat bootprep list-vars --recipe-version 23.11.1
This command outputs a table showing the variables and their values from the
product_vars.yaml file in the appropriate branch of the
hpc-csm-software-recipe repository in VCS.
By default, the sat bootprep run command uses the variables from the latest
installed version of the HPC CSM Software Recipe. Override this with the
--recipe-version command line argument to sat bootprep run.
(ncn-m001#) For example, to explicitly select the 23.11.1 version of the HPC
CSM Software Recipe default variables, specify --recipe-version 23.11.1:
sat bootprep run --recipe-version 23.11.1 compute-and-uan-bootprep.yaml
The entire sat bootprep input file is not rendered by the Jinja2 template
engine. Jinja2 template rendering of the input file is performed individually
for each supported value. The values of the following keys in the bootprep
input file support rendering as a Jinja2 template and thus support variables:
name key of each configuration under the configurations key.layers key in a
configuration:
nameplaybookgit.branchproduct.versionproduct.branchimages key:
namebase.product.versionbase.product.filter.archbase.product.filter.prefixbase.product.filter.wildcardconfigurationsession_templates key:
nameconfigurationbos_parameters.boot_sets.BOOT_SET_NAME.rootfs_provider_passthroughJinja2 built-in filters may be used in values of any of the keys listed above. (NOTE: When the value of a key in the bootprep input file is a Jinja2 expression, it must be quoted to pass YAML syntax checking.)
In addition, Python string methods can be called on the string variables.
Variable names with hyphens are not allowed in Jinja2 expressions because they
are parsed as an arithmetic expression instead of a single variable. To support
product names with hyphens, sat bootprep converts hyphens to underscores in
all top-level keys of the default HPC CSM Software Recipe variables. It also
converts any variables sourced from the command line or any variable files
provided directly. When referring to a variable with hyphens in the bootprep
input file, keep this in mind. For example, to refer to the product version
variable for slingshot-host-software in the bootprep input file, write
"{{slingshot_host_software.version}}".
The following example bootprep input file shows how a variable of a USS version can be used in an input file that creates a CFS configuration for computes. Only one layer is shown for brevity.
---
configurations:
- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
layers:
- name: uss-compute-{{uss.working_branch}}
playbook: cos-compute.yml
product:
name: uss
version: "{{uss.version}}"
branch: "{{uss.working_branch}}"
NOTE: When the value of a key in the bootprep input file is a Jinja2 expression, it must be quoted to pass YAML syntax checking.
Jinja2 expressions can also use filters and Python’s built-in string methods to
manipulate the variable values. For example, suppose only the major and minor
components of a USS version are to be used in the branch name for the USS
layer of the CFS configuration. Use the split string method to
achieve this as follows:
---
configurations:
- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
layers:
- name: uss-compute-{{uss.working_branch}}
playbook: cos-compute.yml
product:
name: uss
version: "{{uss.version}}"
branch: integration-{{uss.version.split('.')[0]}}-{{uss.version.split('.')[1]}}
Additional variables are available besides the variables provided by the
--vars option, the --vars-file option, and the HPC CSM Software Recipe.
These additional variables are dynamic because their values are determined at
run-time based on the context in which they appear. Available dynamic variables
include the following:
The variable base.name can be used in the name of an image under the
images key. The value of this variable is the name of the IMS image or
recipe used as the base of this image.
The variable image.name can be used in the name of a session template
under the session_templates key. The value of this variable is the name of
the IMS image used in this session template.
NOTE: The name of a session template is restricted to 45 characters. Keep
this in mind when using image.name in the name of a session template.
These variables reduce the need to duplicate values throughout the sat bootprep input file and make the following use cases possible:
bootprep input filesThis section provides an example bootprep input file. It also gives instructions for obtaining the default bootprep input files delivered with a release of the HPC CSM Software Recipe.
bootprep input fileThe following bootprep input file provides an example of using most of the features described in previous sections. It is not intended to be a complete bootprep file for the entire CSM product.
---
configurations:
- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
layers:
- name: uss-compute-{{uss.working_branch}}
playbook: uss-compute.yml
product:
name: uss
version: "{{uss.version}}"
branch: "{{uss.working_branch}}"
- name: cpe-pe_deploy-{{cpe.working_branch}}
playbook: pe_deploy.yml
product:
name: cpe
version: "{{cpe.version}}"
branch: "{{cpe.working_branch}}"
images:
- name: "{{default.note}}{{base.name}}{{default.suffix}}"
ref_name: base_uss_image
base:
product:
name: uss
type: recipe
version: "{{uss.version}}"
- name: "compute-{{base.name}}"
ref_name: compute_image
base:
image_ref: base_uss_image
configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
configuration_group_names:
- Compute
session_templates:
- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
image:
image_ref: compute_image
configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
bos_parameters:
boot_sets:
compute:
kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
node_roles_groups:
- Compute
rootfs_provider_passthrough: "sbps:v1:iqn.2023-06.csm.iscsi:_sbps-hsn._tcp.{{default.system-name}}.{{default.site-domain}}:300"
(NOTE: Make sure to define the values for system-nameand site-domain in the site-vars.yaml file.)
bootprep input filesDefault bootprep input files are delivered by the HPC CSM Software Recipe product. Access these
files by cloning the hpc-csm-software-recipe repository, as described in
Accessing sat bootprep files.
(ncn-m001#) Find the default input files in the bootprep directory of the
cloned repository:
ls bootprep/
bootprep input fileThe sat bootprep generate-example command was not updated for
recent bootprep schema changes. It is recommended to instead use the
default bootprep input files described in Access Default Bootprep Input
Files. The sat bootprep generate-example command will be updated in a future release of SAT.
bootprep resultsThe sat bootprep run command uses information from the bootprep input file to
create CFS configurations, IMS images, and BOS session templates. For easy
reference, the command also includes output summarizing the final creation
results.
(ncn-m001#) Here is a sample table output after running sat bootprep run:
################################################################################
CFS configurations
################################################################################
+------------------+
| name |
+------------------+
| example-config-1 |
| example-config-2 |
+------------------+
################################################################################
IMS images
################################################################################
+---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
| name | preconfigured_image_id | final_image_id | configuration | configuration_group_names |
+---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
| example-image | c1bcaf00-109d-470f-b665-e7b37dedb62f | a22fb912-22be-449b-a51b-081af2d7aff6 | example-config | Compute |
+---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
################################################################################
BOS session templates
################################################################################
+------------------+----------------+
| name | configuration |
+------------------+----------------+
| example-template | example-config |
+------------------+----------------+
The sat bootprep run command uses information from the bootprep input files to
create CFS configurations, IMS images, and BOS session templates. To restrict
this creation into separate stages, use the --limit option and list whether to
create configurations, images, session_templates, or some combination of
these. IUF uses the --limit option in this way to install, upgrade, and
deploy products on a system in stages.
(ncn-m001#) For example, to create only CFS configurations, use --limit configurations as follows:
sat bootprep run --limit configurations example-bootprep-input-file.yaml
Example output:
INFO: Validating given input file example-bootprep-input-file.yaml
INFO: Input file successfully validated against schema
INFO: Creating 3 CFS configurations
...
INFO: Skipping creation of IMS images based on value of --limit option.
INFO: Skipping creation of BOS session templates based on value of --limit option.
(ncn-m001#) To create only IMS images and BOS session templates, use --limit images and --limit session-templates as follows:
sat bootprep run --limit images --limit session_templates example-bootprep-input-file.yaml
Example output:
INFO: Validating given input file example-bootprep-input-file.yaml
INFO: Input file successfully validated against schema
INFO: Skipping creation of CFS configurations based on value of --limit option.
The following is an example of how to use the if_exists key on an image. This
key can be used to specify the action taken when an item (a configuration,
image, or session template) has a name that conflicts with an item that already
exists on the system.
if_exists options
skip skip the creation of the image in the bootprep file and keep
the existing imageoverwrite overwrite the existing image with the image in the
bootprep fileabort abort the sat bootprepExample image:
images:
- name: example-compute-image
if_exists: 'skip' # options: 'skip' | 'overwrite' | 'abort'
description: >
An example compute node image built from an existing IMS recipe.
base:
ims:
name: example-compute-image-recipe
type: recipe
configuration: example-compute-config
configuration_group_names:
- Compute
This key works for configurations and session-templates in addition to images as
shown in the example above.
bootprep schemaThe contents of the YAML input files used by sat bootprep must conform to a
schema which defines the structure of the data. The schema definition is written
using the JSON Schema format. (Although the format is named “JSON Schema”, the
schema itself is written in YAML as well.) More information, including introductory
materials and a formal specification of the JSON Schema metaschema, can be found
on the JSON Schema website.
(ncn-m001#) To view the exact schema specification, use sat bootprep view-schema.
sat bootprep view-schema
---
$schema: "https://json-schema.org/draft/2020-12/schema"
Beginning of example output:
title: Bootprep Input File
description: >
A description of the set of CFS configurations to create, the set of IMS
images to create and optionally customize with the defined CFS configurations,
and the set of BOS session templates to create that reference the defined
images and configurations.
type: object
additionalProperties: false
The raw schema definition can be difficult to understand without experience
working with JSON Schema specifications. For this reason, a feature is included
with sat bootprep that generates user-friendly HTML documentation for the input
file schema. This HTML documentation can be browsed with a web browser.
(ncn-m001#) Create a documentation tarball using sat bootprep.
sat bootprep generate-docs
Example output:
INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
An alternate output directory can be specified with the --output-dir
option. The generated tarball is always named bootprep-schema-docs.tar.gz.
sat bootprep generate-docs --output-dir /tmp
Example output:
INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
(user@hostname>) From another machine, copy the tarball to a local directory.
scp root@ncn-m001:bootprep-schema-docs.tar.gz .
(user@hostname>) Extract the contents of the tarball and open the contained index.html.
tar xzvf bootprep-schema-docs.tar.gz
Example output:
x bootprep-schema-docs/
x bootprep-schema-docs/index.html
x bootprep-schema-docs/schema_doc.css
x bootprep-schema-docs/schema_doc.min.js
another-machine$ open bootprep-schema-docs/index.html