The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components.
SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands
used on the Cray XC platform. For more information on SAT commands, see System Admin Toolkit Command Overview.
Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and are reported through Redfish.
In CSM 1.3 and newer, the sat command is automatically available on all the
Kubernetes NCNs. For more information, see SAT in CSM. Older
versions of CSM do not have the sat command automatically available, and SAT
must be installed as a separate product.
Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides instruction on the SAT Container Environment.
The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the
HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are
similarities between SAT commands and xt commands used on the Cray XC platform.
The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each have their own set of options.
The sat command-line utility runs in a container using Podman, a daemonless container runtime. SAT runs on
Kubernetes manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
container is launched, executes the command, and upon the command’s completion the container exits. The following two
examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
(CONTAINER-ID)sat-container# sat status
ncn-m001# sat status
Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
example.
ncn-m001# sat-man status
A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
ncn-m001# man sat-podman
The host name in a command prompt indicates where the command must be run. The account that must run the command is also indicated in the prompt.
root or super-user account always has the # character at the end of the prompt and has the host name of the
host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
referred to as user.| Command Prompt | Meaning |
|---|---|
ncn-m001# |
Run on one of the Kubernetes Manager servers. (Non-interactive) |
(CONTAINER_ID) sat-container# |
Run the command inside the SAT container environment by first running sat bash. (Interactive) |
Examples of the sat status command used by an administrator:
ncn-m001# sat status
ncn-m001# sat bash
(CONTAINER_ID) sat-container# sat status
In CSM 1.3 and newer, the sat command is automatically available on all the Kubernetes NCNs, but it is still possible
to install SAT as a separate product stream. Any version of SAT installed as a separate product stream overrides the
sat command available in CSM. Installing the SAT product stream allows additional supporting components to be added:
An entry for SAT in the cray-product-catalog Kubernetes ConfigMap is only created by installing the SAT product
stream. Otherwise, there will be no entry for this version of SAT in the output of sat showrev.
The sat-install-utility container image is only available with the full SAT product stream. This container image
provides uninstall and activate functionality when used with the prodmgr command. (In SAT 2.3 and older, SAT was
only available to install as a separate product stream. Because these versions were packaged with
sat-install-utility, it is still possible to uninstall these versions of SAT.)
The docs-sat RPM package is only available with the full SAT product stream.
The sat-config-management git repository in Gitea (VCS) and thus the SAT layer of NCN CFS configuration is
only available with the full SAT product stream.
If the SAT product stream is not installed, there will be no configuration content for SAT in VCS. Therefore, CFS
configurations that apply to NCNs (for example, ncn-personalization) should not include a SAT layer.
The SAT configuration layer modifies the permissions of files left over from prior installations of SAT, so that the
Keycloak username that authenticates to the API gateway cannot be read by users other than root. Specifically, it
it does the following:
Modifies the sat.toml configuration file which contains the username so that it is only readable by root.
Modifies the /root/.config/sat/tokens directory so that the directory is only readable by root. This is needed
because the names of the files within the tokens directory contain the username.
Regardless of the SAT configuration being applied, passwords and the contents of the tokens are never readable by other users. These permission changes only apply to files created by previous installations of SAT. In the current version of SAT all files and directories are created with the appropriate permissions.
Most sat subcommands depend on services or components from other products in the
HPE Cray EX (Shasta) software stack. The following list shows these dependencies
for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
sat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nid