The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components.
SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands
used on the Cray XC platform. For more information on SAT commands, see SAT Command Overview.
Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and are reported through Redfish.
In CSM 1.3 and newer, the sat command is automatically available on all the
Kubernetes NCNs. For more information, see SAT in CSM. Older
versions of CSM do not have the sat command automatically available, and SAT
must be installed as a separate product.
Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides instruction on the SAT Container Environment.
The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the
HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are
similarities between SAT commands and xt commands used on the Cray XC platform.
The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each have their own set of options.
The sat command-line utility runs in a container using Podman, a daemonless container runtime. SAT runs on
Kubernetes manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
container is launched, executes the command, and upon the command’s completion the container exits. The following two
examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
(CONTAINER-ID)sat-container# sat status
ncn-m001# sat status
Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
example.
ncn-m001# sat-man status
A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
ncn-m001# man sat-podman
Some SAT subcommands make requests to the Shasta services through the API
gateway and thus require authentication to the API gateway in order to function.
Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
be configured. In order to use the SAT S3 bucket, the System Administrator must
generate the S3 access key and secret keys and write them to a local file. This
must be done on every Kubernetes manager node where SAT commands are run.
For more information on authentication requests, see System Security and Authentication in the Cray System Management Documentation. The following is a table describing SAT commands and the types of authentication they require.
| SAT Subcommand | Authentication/Credentials Required | Man Page | Description |
|---|---|---|---|
sat auth |
Responsible for authenticating to the API gateway and storing a token. | sat-auth |
Authenticate to the API gateway and save the token. |
sat bmccreds |
Requires authentication to the API gateway. | sat-bmccreds |
Set BMC passwords. |
sat bootprep |
Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is done on ncn-m001 during the install. |
sat-bootprep |
Prepare to boot nodes with images and configurations. |
sat bootsys |
Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
sat-bootsys |
Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. |
sat diag |
Requires authentication to the API gateway. | sat-diag |
Launch diagnostics on the HSN switches and generate a report. |
sat firmware |
Requires authentication to the API gateway. | sat-firmware |
Report firmware version. |
sat hwhist |
Requires authentication to the API gateway. | sat-hwhist |
Report hardware component history. |
sat hwinv |
Requires authentication to the API gateway. | sat-hwinv |
Give a listing of the hardware of the HPE Cray EX system. |
sat hwmatch |
Requires authentication to the API gateway. | sat-hwmatch |
Report hardware mismatches. |
sat init |
None | sat-init |
Create a default SAT configuration file. |
sat jobstat |
Requires authentication to the API gateway. | sat-jobstat |
Check the status of jobs and applications. |
sat k8s |
Requires Kubernetes configuration and authentication, which is automatically configured on ncn-m001 during the install. |
sat-k8s |
Report on Kubernetes replica sets that have co-located (on the same node) replicas. |
sat linkhealth |
This command has been deprecated. | ||
sat nid2xname |
Requires authentication to the API gateway. | sat-nid2xname |
Translate node IDs to node XNames. |
sat sensors |
Requires authentication to the API gateway. | sat-sensors |
Report current sensor data. |
sat setrev |
Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | sat-setrev |
Set HPE Cray EX system revision information. |
sat showrev |
Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | sat-showrev |
Print revision information for the HPE Cray EX system. |
sat slscheck |
Requires authentication to the API gateway. | sat-slscheck |
Perform a cross-check between SLS and HSM. |
sat status |
Requires authentication to the API gateway. | sat-status |
Report node status across the HPE Cray EX system. |
sat swap |
Requires authentication to the API gateway. | sat-swap |
Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. |
sat xname2nid |
Requires authentication to the API gateway. | sat-xname2nid |
Translate node and node BMC XNames to node IDs. |
sat switch |
This command has been deprecated. It has been replaced by sat swap. |
In order to authenticate to the API gateway, you must run the sat auth
command. This command will prompt for a password on the command line. The
username value is obtained from the following locations, in order of higher
precedence to lower precedence:
--username global command-line option.username option in the api_gateway section of the config file at
~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file
will be obtained and saved to ~/.config/sat/tokens. Subsequent sat commands
will determine the username the same way as sat auth described above and will
use the token for that username if it has been obtained and saved by sat auth.
The host name in a command prompt indicates where the command must be run. The account that must run the command is also indicated in the prompt.
root or super-user account always has the # character at the end of the prompt and has the host name of the
host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
referred to as user.| Command Prompt | Meaning |
|---|---|
ncn-m001# |
Run on one of the Kubernetes Manager servers. (Non-interactive) |
(CONTAINER_ID) sat-container# |
Run the command inside the SAT container environment by first running sat bash. (Interactive) |
Here are examples of the sat status command used by an administrator.
ncn-m001# sat status
ncn-m001# sat bash
(CONTAINER_ID) sat-container# sat status
In CSM 1.3 and newer, the sat command is automatically available on all the Kubernetes NCNs, but it is still possible
to install SAT as a separate product stream. Any version of SAT installed as a separate product stream overrides the
sat command available in CSM. Installing the SAT product stream allows additional supporting components to be added:
An entry for SAT in the cray-product-catalog Kubernetes ConfigMap is only created by installing the SAT product
stream. Otherwise, there will be no entry for this version of SAT in the output of sat showrev.
The sat-install-utility container image is only available with the full SAT product stream. This container image
provides uninstall and downgrade functionality when used with the prodmgr command. (In SAT 2.3 and older, SAT was
only available to install as a separate product stream. Because these versions were packaged with
sat-install-utility, it is still possible to uninstall these versions of SAT.)
The docs-sat RPM package is only available with the full SAT product stream.
The sat-config-management git repository in Gitea (VCS) and thus the SAT layer of NCN CFS configuration is
only available with the full SAT product stream.
If the SAT product stream is not installed, there will be no configuration content for SAT in VCS. Therefore, CFS
configurations that apply to management NCNs (for example, management-23.5.0) should not include a SAT layer.
The SAT configuration layer modifies the permissions of files left over from prior installations of SAT, so that the
Keycloak username that authenticates to the API gateway cannot be read by users other than root. Specifically, it
it does the following:
Modifies the sat.toml configuration file which contains the username so that it is only readable by root.
Modifies the /root/.config/sat/tokens directory so that the directory is only readable by root. This is needed
because the names of the files within the tokens directory contain the username.
Regardless of the SAT configuration being applied, passwords and the contents of the tokens are never readable by other users. These permission changes only apply to files created by previous installations of SAT. In the current version of SAT all files and directories are created with the appropriate permissions.
Most sat subcommands depend on services or components from other products in the
HPE Cray EX (Shasta) software stack. The following list shows these dependencies
for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
sat jobstatsat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nid