Cray System Management
v
1.6
1.5
1.4
1.3
1.2
1.0
0.9
Cray System Management (CSM) 1.6 Release Notes
Cray System Management (CSM) 1.6.1 Release Notes
Cray System Management (CSM) 1.6.2 Release Notes
Cray System Management (CSM) Administration Guide
System Admin Toolkit (SAT)
About SAT
Introduction to SAT
SAT Command Overview
SAT Dependencies
SAT in CSM
SAT Uninstall and Downgrade
SAT Usage
Change the BOS Version
Change the CFS Version
SAT Bootprep
SAT and IUF
SAT on an External System
SAT Configuration
Authenticate SAT Commands
Configure Multi-tenancy (Optional)
Generate SAT S3 Credentials
Set System Revision Information
Create a Backup of HMS Items for reinstall
Component Names (xnames)
Restore HSM
Validate CSM Health
Configure the Cray Command Line Interface (cray CLI)
utility storage
Adding a Ceph Node to the Ceph Cluster
Add Ceph OSDs
Adjust Ceph Pool Quotas
Alternate Storage Pools
Ceph Daemon Memory Profiling
Ceph Deep Scrubs
Ceph Health States
Ceph Orchestrator Usage
Ceph Service Check Script Usage
Ceph Storage Types
ceph-upgrade-tool.py Usage
Cephadm Reference Material
Collect Information about the Ceph Cluster
Dump Ceph Crash Data
Identify Ceph Latency Issues
Manage Ceph Services
Shrink the Ceph Cluster
Restore Nexus Data After Data Corruption
Shrink Ceph OSDs
Troubleshoot Ceph-Mon Processes Stopping and Exceeding Max Restarts
Troubleshoot Ceph MDS Client Connectivity Issues
Troubleshooting Ceph MDS Reporting Slow Requests and Failure on Client
Troubleshoot Ceph New RGW Deployment Failing
Troubleshoot Ceph OSDs Not Being Created on Disks
Troubleshoot Ceph OSDs Reporting Full
Troubleshoot Ceph Services Not Starting After a Server Crash
Troubleshoot Failure to Get Ceph Health
Troubleshoot HEALTH ERR Module devicehealth has failed table Device already exists
Troubleshoot Insufficient Standby MDS Daemons Available
Troubleshoot Large Object Map Objects in Ceph Health
Troubleshoot Pods Failing to Restart on Other Worker Nodes
Fixing incorrect number of PG Issues
Troubleshoot if RGW Health Check Fails
Troubleshoot S3FS Mount Issues
Troubleshoot System Clock Skew
Troubleshoot a Down OSD
Troubleshoot an Unresponsive Rados-Gateway (radosgw) S3 Endpoint
Troubleshoot Ceph image with tag'<none>'
Utility Storage
Update ceph node-exporter config to monitor SNMP counters
network
Management Network User Guide
Manual Switch Configuration
Fresh Install
Added Hardware
Generate Switch Configurations
Apply Custom Switch Configuration CSM 1.2
Apply Switch Configurations
CSM Automatic Network Utility
CANU Installation
Troubleshoot CANU Validation Errors
Use CANU to Verify, Generate, or Compare Switch Configurations
Generate Switch Configs Including Custom Configurations
Initializing CANU
Introduction to CANU
Quick start guide to CANU
Uninstall CANU
Update CANU From CSM Release Tarball
Use CANU to Generate Full Network Configuration
Dell Installation and Configuration Guide
Configure Access Control Links (ACLs)
Configure Address Resolution Protocol (ARP)
Back Up a Switch Configuration
Configure Domain Name System (DNS) Client
Configure Domain Name
Configure Hostnames
Configure Internet Group Multicast Protocol (IGMP)
Configure Link Aggregation Group (LAG)
Link layer discovery protocol (LLDP)
Configure Locator LED
Configure Loopback Interface
Configure Management Interface
Configure Multiple Spanning Tree Protocol (MSTP)
Network Time Protocol (NTP) Client
Configure Physical Interfaces
Configure QoS
Configure Remote Logging
Reset Dell Switch Configuration
Configure SNMPv2c community
Dell SNMPv3 Users
Configure Secure Shell (SSH)
Configure System Images
Perform an Upgrade on Dell Switches
Configure Virtual Local Access Networks (VLANs)
Configure VLAN Interface
VLAN Trunking 802.1Q
Using canu-inventory with Ansible
Upgrade CANU
Collect Data
Configuration Management
Configuring SNMP in CSM
Mellanox Installation and Configuration Guide
Access control lists (ACLs)
Address resolution protocol (ARP)
Backing up switch configuration
BGP basics
Cable diagnostics
Check BGP and MetalLB
Check current DHCP leases
Check DHCP lease is getting allocated
Check HSM
Check KEA DHCP logs
Computes/UANs/Application Nodes
Large Number of DHCP Declines During a Node Boot
Domain name system (DNS) client
Domain name
You are getting an IP address, but not the correct one. Duplicate IP address check
Exec banners
Hostname
IGMP
Ip filter
Key features used in the management network configuration
Link aggregation group (LAG)
Large
Link layer discovery protocol (LLDP)
Loopback interface
Management interface
Example of how to configure Scenario A or B
Management network functions in detail
Medium
Multi-chassis interface
MLAG (Multi-Chassis LAG)
MLAG
Multiple spanning tree protocol (MSTP)
Native VLAN
TCPDUMP
NCNs on Install
Network types – Naming and segment Function
Network traffic pattern inside of the system
Network Time Protocol (NTP) Client
Open shortest path first (OSPF) v2
Physical interfaces
PIM-SM bootstrap router (BSR) and rendezvous-point (RP)
Rebooting NCN and PXE fails
Remote logging
How to connect management network to your campus network
Routed interfaces
Scenario A network connection via management network
Scenario B network connection via high speed network
Small
SNMPv2c community
Mellanox SNMPv3 users
Spine-leaf architecture
Spine-leaf architecture
Why are spine-leaf architectures becoming more popular?
Secure shell (SSH)
Mac address Table
Static routing
Confirm the status of the cray-dhcp-kea pods/services
System images
Test TFTP traffic (Aruba Only)
Typical configuration of MLAG link connecting to NCN
Typical configuration of MLAG between switches
Performing Upgrade On Mellanox Switches
Verify the switches are forwarding DHCP traffic
Verify BGP
Verify the DHCP traffic on the workers
Verify route to TFTP
Very Large (Exascale)
Virtual local access networks (VLANs)
VLAN interface
VLAN trunking 802.1Q
Web user interface (WebUI)
Aruba Installation and Configuration Guide
802.1X
Access Control Lists (ACLs)
Address Resolution Protocol (ARP)
Backup a Switch Configuration
Border Gateway Protocol (BGP) Basics
Bluetooth Capabilities
Cable Diagnostics
Check BGP and MetalLB
Check Current DHCP Leases
Check DHCP Lease is Getting Allocated
Check HSM
Check KEA DHCP Logs
Classifier Policies
Verify Computes/UANs/Application Nodes
Large Number of DHCP Declines During a Node Boot
Configure Domain Name Service (DNS) Clients
Configure Domain Names
Check for Duplicate IP Addresses
Configure Exec Banners
Configure Hostnames
Configure Internet Group Multicast Protocol (IGMP)
Initial Prioritization
Introduction
Key Features Used in the Management Network Configuration
Link Aggregation Group (LAG)
Link Layer Discovery Protocol (LLDP)
Locator LED
Loopback Interface
MAC Authentication
Management Interface
Example of How to Configure Scenario A or B
System Management Network Functions
VSX ISL HA
VSX MCLAG Link HA
VSX Member Power Failure
VSX Split
Multi-Chassis Link Aggregation Group (MCLAG)
Message-Of-The-Day (MOTD)
Multicast Source Discovery Protocol (MSDP)
Multiple Spanning Tree Protocol (MSTP)
Native VLAN
NCN tcpdump
NCNs on Install
Network Types – Naming and Segment Function
Network Topologies
Network Traffic Pattern
Notices
Network Time Protocol (NTP) Client
Open Shortest Path First (OSPF) v2
Physical Interfaces
PIM-SM Bootstrap Router (BSR) and Rendezvous Point (RP)
Port Mirroring
Port Security
Queuing and Scheduling
RADIUS
Rebooting NCNs and PXE Fails
Redundant Power Supplies
Remote Logging
Connect the Management Network to a Campus Network
Routed interfaces
Scenario A Network Connection via Management Network
Scenario B Network Connection via High-Speed Network
Simple Network Management Protocol (SNMP) Agent
SNMPv2c Community
SNMP traps
Aruba SNMPv3 Users
Spine-Leaf Architecture
Spine-leaf Architecture
Secure Shell (SSH)
Static Routing
Confirm the Status of the cray-dhcp-kea Pods
TACACS
Test TFTP Traffic (Aruba Only)
Typical Configuration of VSX
Typical Edge Port Configuration
Typical Configuration of MCLAG Link
Unidirectional Link Detection (UDLD)
Perform a VSX Upgrade on Aruba Switches
Verify the Switches are Forwarding DHCP Traffic
Verify BGP
Verify the DHCP Traffic on the Worker Nodes
Verify Route to TFTP
Virtual Local Access Networks (VLANs)
VLAN Interface
VLAN Trunking 802.1Q
Virtual Switching Framework (VSF) - 6300 Only
Virtual Switching Extension (VSX)
What is VSX?
Switch Replacement in the VSX Cluster
VSX Sync
Web User Interface (WebUI)
Erase All zeroize
Edge switch cabling guide
External User Guides
Network Tests
Reinstall
Replace Switch
Save a Configuration
Prometheus SNMP Exporter
Transceivers and Cables
Example of the Connections Used in Shasta Management Network
Validate Cabling
Validate the SHCD
Validate Switch Configurations
Wipe Management Switch Configuration
Aruba splitting of QSFP+ and QSFP28 ports
Backup a Custom Configuration
BICAN Support Matrix - Shasta Customer Access Networks
BICAN switch configuration
Bifurcating the CAN - Feature Details
BICAN Summary
Bonded UAN Configuration
Cable Management Network Servers
firmware
Update Management Network Firmware
hardware
EX2500 Installation and Cabling
Access to System Management Services
Connect to Switch over USB-Serial Cable
Connect to the HPE Cray EX Environment
Create a CSM Configuration Upgrade Plan
Default IP Address Ranges
Gateway Testing
Network
customer accessible networks
Connect to the CMN and CAN
Customer Access Networks
network
Enabling Customer High Speed Network Routing
Management Network Upgrade CSM 1.2 to 1.3
scripts
sls
sls utils Library
Customer Accessible Networks
CAN/CMN with Dual-Spine Configuration
Externally Exposed Services
Troubleshoot CMN issues
BI-CAN Aruba/Arista Configuration
MetalLB Peering with Arista Edge Router
dhcp
DHCP boot file customization
DHCP
Troubleshoot DHCP Issues
dns
Domain Name Service (DNS) Overview
Enable ncsd on UANs
Manage the DNS Unbound Resolver
PowerDNS Configuration
PowerDNS Migration Guide
Troubleshoot Common DNS Issues
Troubleshoot PowerDNS
external dns
External DNS
External DNS Failing to Discover Services Workaround
External DNS CSI Input Values
Ingress Routing
Troubleshoot DNS Configuration Issues
Troubleshoot Connectivity to Services with External IP addresses
Update the cmn-external-dns value post-installation
metallb bgp
Check BGP Status and Reset Sessions
MetalLB Configuration
MetalLB in BGP-Mode
Troubleshoot BGP not Accepting Routes from MetalLB
Troubleshoot Services without an Allocated IP Address
power management
Cray Advanced Platform Monitoring and Control (CAPMC)
Ignore Nodes with CAPMC
Liquid-cooled Node Power Management
Power Off Compute Cabinets
Power Off Management Cabinets
Power Off Storage Cabinets
Power Off the External Lustre File System
Power On Compute Cabinets
Power On and Boot Managed Nodes
Power On and Start the Management Kubernetes Cluster
Power On the External Lustre File System
Prepare the System for Power Off
Recover from a Liquid Cooled Cabinet EPO Event
Save Management Network Switch Configuration Settings
Set the Turbo Boost Limit
Shut Down and Power Off Managed Nodes
Shut Down and Power Off the Management Kubernetes Cluster
Standard Rack Node Power Management
System Power Off Procedures
System Power On Procedures
User Access to Compute Node Power Data
Power Management
Power Control Service
Node Card Power Management
Power Control Service (PCS)
Recover from a Liquid Cooled Cabinet EPO Event
CSM product management
Change Passwords and Credentials
Configure CSM packages with CFS
Configure Keycloak Account
Configure the root password and SSH keys in Vault
Post-Install Customizations
Redeploying a Chart
Remove Artifacts from Product Installations
Set up passwordless SSH
Validate Signed RPMs
hmcollector
Adjust HM Collector Ingress Replicas and Resource Limits
iscsi sbps
iSCSI SBPS (Scalable Boot Content Projection Service)
observability
Install and Upgrade Observability Framework
security and authentication
API Authorization
Access the Keycloak User Management UI
Add LDAP User Federation
Add Root Service Account for Gigabyte Controllers
Audit Logs
Authenticate an Account with the Command Line
Backup and Restore Vault Clusters
Certificate Types
Change Air-Cooled Node BMC Credentials Using SAT
Change Credentials on ServerTech PDUs
Change Cray EX Liquid-Cooled Cabinet Global Default Password
Change the Keycloak Token Lifetime
Set NCN Image Root Password, SSH Keys, and Timezone
Set NCN Image Root Password, SSH Keys, and Timezone on PIT Node
Change Root Passwords for Compute Nodes
Change the Keycloak Admin Password
Change the LDAP Server IP Address for Existing LDAP Server Content
Change the LDAP Server IP Address for New LDAP Server Content
Configure Keycloak for LDAP/AD authentication
Configure root user on HPE iLO BMCs
Configure the RSA Plugin in Keycloak
Create Internal Groups in the Keycloak Shasta Realm
Create Internal User Accounts in the Keycloak Shasta Realm
Create a Backup of the Keycloak Postgres Database
Create a Service Account in Keycloak
Default Keycloak Realms, Accounts, and Clients
Delete Internal User Accounts in the Keycloak Shasta Realm
Get a Long-Lived Token for a Service Account
HashiCorp Vault
Keycloak Operations
Keycloak Service Recovery
Keycloak User Localization
Keycloak User Management with kcadm.sh
Make HTTPS Requests from Sources Outside the Management Kubernetes Cluster
Manage Sealed Secrets
Manage System Passwords
PKI Certificate Authority (CA)
PKI Services
Preserve Username Capitalization for Users Exported from Keycloak
Provisioning a Liquid-Cooled EX Cabinet CEC with Default Credentials
Public Key Infrastructure (PKI)
Recovering from Mismatched BMC Credentials
Remove Internal Groups from the Keycloak Shasta Realm
Remove the Email Mapper from the LDAP User Federation
Remove the LDAP User Federation from Keycloak
Re-Sync Keycloak Users to Compute Nodes
Retrieve an Authentication Token
Retrieve the Client Secret for Service Accounts
SOPS Introduction
Update NCN User SSH Keys
System Security and Authentication
Transport Layer Security (TLS) for Ingress Services
Troubleshoot Common Vault Cluster Issues
Troubleshoot Kyverno configuration manually
Update Default Air-Cooled BMC and Leaf-BMC Switch SNMP Credentials
Update Default ServerTech PDU Credentials used by the Redfish Translation Service (RTS)
Set NCN User Passwords
Updating the Liquid-Cooled EX Cabinet CEC with Default Credentials after a CEC Password Change
Vault Service Recovery
system layout service
Add Liquid-Cooled Cabinets to SLS
Add UAN CAN IP Addresses to SLS
Add an alias to a service
Create a Backup of the SLS Postgres Database
Dump SLS Information
Load SLS Database with Dump File
Restore SLS Postgres Database from Backup
Restore SLS Postgres without an Existing Backup
System Layout Service (SLS)
Update SLS with UAN Aliases
boot orchestration
Boot Orchestration
BOS Services
BOS Workflows
Compute Node Boot Issue Symptom Node Console or Logs Indicate that the Server Response has Timed Out
Boot Issue Symptom Node HSN Interface Does Not Appear or Show Detected Links Detected
Boot Orchestration
Boot UANs
BOS Commands Cheat Sheet
Component Status
BOS Components
Compute Node Boot Issue Symptom Duplicate Address Warnings and Declined DHCP Offers in Logs
Compute Node Boot Issue Symptom Message About Invalid EEPROM Checksum in Node Console or Log
Compute Node Boot Issue Symptom Node is Not Able to Download the Required Artifacts
Compute Node Boot Sequence
Create a Session Template to Boot Compute Nodes with CPS
Create a Session Template to Boot Compute Nodes with Scalable Boot Projection Service (SBPS)
Customize iPXE Binary Names
Determine Which BOS Session Booted a Node
Edit the iPXE Embedded Boot Script
Exporting and Importing BOS Data
Exporting and Importing BSS Data
Healthy Compute Node Boot Process
Kernel Boot Parameters
Limit the Scope of a BOS Session
Log File Locations and Ports Used in Compute Node Boot Troubleshooting
Manage a BOS Session
Manage a Session Template
Multi-tenancy with BOS
Node Boot Root Cause Analysis
BOS Options
Redeploy the iPXE and TFTP Services
Rolling Upgrades using BOS
BOS Session Templates
BOS Sessions
Staging Changes with BOS
Tools for Resolving Compute Node Boot Issues
Troubleshoot Compute Node Boot Issues Related to Dynamic Host Configuration Protocol (DHCP)
Troubleshoot Compute Node Boot Issues Related to Trivial File Transfer Protocol (TFTP)
Troubleshoot Compute Node Boot Issues Related to Unified Extensible Firmware Interface (UEFI)
Troubleshoot Compute Node Boot Issues Related to the Boot Script Service (BSS)
Troubleshoot Compute Node Boot Issues Using Kubernetes
Troubleshoot UAN Boot Issues
Upload Node Boot Information to Boot Script Service (BSS)
View the Status of a BOS Session
resiliency
Recreate StatefulSet Pods on Another Node
Resilience of System Management Services
Resiliency
Resiliency Testing Procedure
Restore System Functionality if a Kubernetes Worker Node is Down
iuf
Install and Upgrade Framework
stages
deliver-product
deploy-product
managed-nodes-rollout
management-nodes-rollout
post-install-check
post-install-service-check
pre-install-check
prepare-images
process-media
update-cfs-config
update-vcs-config
workflows
Populate admin directory with files defining site preferences
Backup
Configuration
Configuration of the Slingshot Fabric Manager
Deploy product
Image preparation
Install or upgrade additional products with IUF
Managed rollout
Management rollout
Prepare for the install or upgrade
Product delivery
Perform Slingshot switch and management network switch firmware updates
Upgrade CSM and additional products with IUF
Validate deployment
examples
iuf abort Examples
iuf activity Examples
iuf list-activities Examples
iuf list-stages Examples
iuf restart Examples
iuf resume Examples
iuf run Examples
iuf workflow Examples
configuration management
ARP Cache Tuning Guide
Accessing sat bootprep Files
Adding Additional Inventory
Ansible Execution Environments
Ansible Log Collection
Automatic Configuration Management
Automatic Session Deletion with session ttl
Backup and Restore VCS Data
CFS Commands Cheat Sheet
CFS Components
CFS Configurations
CFS Flow
CFS Global Options
CFS Key Management and Permission Denied Errors
CFS Session Inventory
CFS Sessions
CFS Sources
Change the Ansible Verbosity
Configuration Management
Configure Ansible
Configure HSN NIC Bonding
Create a Node Personalization CFS Session
Create an Image Customization CFS Session
Create and Populate a VCS Configuration Repository
Customize Configuration Values
Differences Between the V2 and V3 CFS APIs
Enable Ansible Profiling
Exporting and Importing CFS Data
Git Operations
Management Node Image Customization
Management Node Personalization
Paging CFS Records
Managing Sensitive Information in (VCS)
Set Limits for a Configuration Session
Specifying Hosts and Groups
Target Ansible Tasks for Image Customization
Track the Status of a Session
Troubleshoot CFS Issues
Troubleshoot Failed CFS Sessions
Troubleshoot CFS Session Failing to Complete
Troubleshoot CFS Sessions Failing to Start
Update a CFS Configuration
Update the Privacy Settings for Gitea Configuration Content Repositories
VCS Administrative User
VCS Branching Strategy
Version Control Service (VCS)
View Configuration Session Logs
Write Ansible Code for CFS
iSCSI SBPS Worker Node Personalization
hpe pdu
HPE PDU Admin Procedures
hardware state manager
Add a Switch to the HSM Database
Add an NCN to the HSM Database
Component Group Members
Component Groups and Partitions
Component Memberships
Component Partition Members
Create a Backup of the HSM Postgres Database
Backup/Restore HSM User Data (Locks, Groups, and Partitions)
HSM Roles and Subroles
Hardware Management Services (HMS) Locking API
Hardware State Manager (HSM)
Hardware State Manager (HSM) State and Flag Fields
Lock and Unlock Management Nodes
Manage Component Groups
Manage Component Partitions
Manage HMS Locks
Restore Hardware State Manager (HSM) Postgres Database from Backup
Restore Hardware State Manager (HSM) Postgres without an Existing Backup
Set BMC Management Roles
image management
Build a New UAN Image Using the Default Recipe
Build an Image Using IMS REST Service
Configure IMS to Use DKMS
Configure IMS to Validate RPMs
Configure a Remote Build Node
Convert TGZ Archives to SquashFS Images
Create UAN Boot Images
Customize an Image Root Using IMS
Delete or Recover Deleted IMS Content
Exporting and Importing IMS Data
Image Job Performance
Image Management
Image Management Workflows
Import an External Image to IMS
Import an NCN Image to IMS
Troubleshoot Issues with Large Images
Troubleshoot Remote Build Node
Troubleshoot Interactions with zypper
Upload and Register an Image Recipe
Working With aarch64 Images
package repository management
Manage Repositories with Nexus
Nexus Configuration
Nexus Deployment
Nexus Export and Restore
Nexus Service Recovery
Nexus Space Cleanup
Package Repository Management
Package Repository Management with Nexus
Repair Blobstore
Repair Yum Repository Metadata
Restrict Admin Privileges in Nexus
Troubleshoot Nexus
argo
Using Argo Workflows
Using the Argo UI
artifact management
Artifact Management
Generate Temporary S3 Credentials
Manage Artifacts with the Cray CLI
Use S3 Libraries and Clients
firmware
FASUpdate Script
FAS Admin Procedures
FAS CLI
Cleaning up FAS Database
FAS Filters
Backup and Restoring FAS Images
Updating Foxconn Paradise Nodes with FAS
FAS Recipes
Update iLO 5 firmware above v2.78
FAS Recipes and Procedures
Firmware Upgrade using SPP on HPE ProLiant Servers
Update Firmware with FAS
Updating BMC Firmware and BIOS for ncn-m001
Updating BMC Firmware and BIOS for NCNs without FAS
Upload BMC Recovery Firmware into TFTP Server
system management health
Access System Management Health Services
Configure Prometheus Email Alert Notifications
Retrieve SMART data from ClusterStor E1000 nodes via Redfish Exporter
Grafana Dashboards by Component
grok-exporter pod status showing as ContainerStatusUnknown Error
prometheus-kafka-adapter errors during installation
Remove Kiali
System Management Health
System Management Health Checks and Alerts
Troubleshoot Grafana Dashboard
Troubleshoot Prometheus Alerts
UAN NODE Exporter
VictoriaMetrics
node management
Access and Update Settings for Replacement NCNs
Removing a Liquid-cooled blade from a System
Removing a Liquid-cooled blade from a System Using SAT
Removing a Standard rack node from a System
Replace a Compute Blade
Replace a Compute Blade Using SAT
Replace a Standard rack node from a System
Replacing Foxconn Username and Passwords in Vault
Add TLS Certificates to BMCs
Repurpose a Compute Node as a UAN
Add a Standard Rack Node
Reset Credentials on Redfish Devices
Add Additional Air-Cooled Cabinets to a System
S3FS Usage and Guidelines for Shasta
Add Additional Liquid-Cooled Cabinets to a System
Set Gigabyte Node BMC to Factory Defaults
Adding a Liquid-cooled Blade to a System
Swap a Compute Blade with a Different System
Adding a Liquid-cooled blade to a System Using SAT
Swap a Compute Blade with a Different System Using SAT
Boot a storage node into new image without upgrading CSM
Switch PXE Boot from Onboard NIC to PCIe
Build NCN Images Locally
TLS Certificates for Redfish BMCs
Change Java Security Settings
Troubleshoot Interfaces with IP Address Issues
Change Settings for HMS Collector Polling of Air-Cooled Nodes
Troubleshoot Issues with Redfish Endpoint Discovery
Check and Set the metal.no-wipe Setting on NCNs
Troubleshoot Loss of Console Connections and Logs on Gigabyte Nodes
Check the BMC Failover Mode
Update Compute Node Mellanox HSN NIC Firmware
Clear Space in Root File System on Worker Nodes
Update the Gigabyte Node BIOS Time
Configuration of NCN Bonding
Update the HPE Node BIOS Time
Configure NTP on NCNs
Updating Cabinet Routes on Management NCNs
Customize PCIe Hardware
Use the Physical KVM
Customize PCIe Hardware
Verify Node Removal
Defragment NID Numbering
View BIOS Logs for Liquid-Cooled Nodes
Disable Nodes
Manual Wipe Procedures
Dump a Non-Compute Node
Clear Gigabyte CMOS
Enable Nodes
Enable Passwordless Connections to Liquid Cooled Node BMCs
Enable IPMI access on HPE iLO BMCs
Find Node Type and Manufacturer
Launch a Virtual KVM on Gigabyte Servers
Launch a Virtual KVM on Intel Servers
Move a Standard Rack Node
Move a Standard Rack Node (Same Rack/Same HSN Ports)
Move a liquid-cooled blade within a System
NCN Drive Identification
NCN NIC Replacement
NCN Network Troubleshooting
Node Management
Node Management Workflows
Reboot NCNs
Add Remove Replace NCNs
Add NCN Data
Alpha Framework to Add, Remove, Replace, or Move NCNs
Add Switch Configuration for NCN
Allocate NCN IP Addresses
Boot NCN
Collect NCN MAC Addresses
Redeploy Services Impacted by Adding or Permanently Removing Storage Nodes
Remove NCN Data
Remove NCN from Role
Remove Switch Configuration for NCN
Update Firmware
Update NCN BIOS TPM State
Validate Health
Validate Added NCN
Rebuild NCNs
Final Validation Steps
Identify Nodes and Update Metadata
Post Rebuild Storage Node Validation
Power Cycle and Rebuild Nodes
Prepare Storage Nodes
Re-Add a Storage Node to Ceph
Rebuild NCNs
Validate Boot Loader
bare metal
Bare-Metal Steps
Fresh Install Setting NodeBMC and RouterBMC Redfish Credentials
conman
Access Compute Node Logs
Access Console Log Data Via the System Monitoring Framework (SMF)
Complete Reset of the Console Services
ConMan
Configure Log Rotation
Console Services Troubleshooting Guide
Disable ConMan After the System Software Installation
Establish a Serial Connection to NCNs
Log in to a Node Using ConMan
Manage Node Consoles
Troubleshoot ConMan Asking for Password on SSH Connection
Troubleshoot ConMan Blocking Access to a Node BMC
Troubleshoot ConMan Failing to Connect to a Console
Troubleshoot Console Node Pod Stuck in Terminating State
kubernetes
About Kubernetes Taints and Labels
Kubernetes Encryption
About Postgres
About etcd
About kubectl
Backups for Etcd Clusters Running in Kubernetes
Kubernetes and Bare Metal EtcD Certificate Renewal
Check for and Clear etcd Cluster Alarms
Check the Health of etcd Clusters
Clear Space in an etcd Cluster Database
Configure kubectl Credentials to Access the Kubernetes APIs
containerd
Create a Manual Backup of Bare-Metal etcd Cluster
Create a Manual Backup of a Healthy etcd Cluster
Determine if Pods are Hitting Resource Limits
Disaster Recovery for Postgres
Fix Failed to start etcd on Master NCN
Increase Kafka Pod Resource Limits
Increase the PVC size in an etcd Cluster Database
Increase Pod Resource Limits
Kubernetes
Kubernetes Networking
Kubernetes Storage
Kyverno policy management
Pod Resource Limits
Rebuild Unhealthy etcd Clusters
Recover from Postgres WAL Event
Repopulate Data in etcd Clusters When Rebuilding Them
Report the Endpoint Status for etcd Clusters
Restore Bare-Metal etcd Clusters from an S3 Snapshot
Restore Postgres
Restore an etcd Cluster from a Backup
Retrieve Cluster Health Information Using Kubernetes
TDS Lower CPU Requests
Troubleshoot Intermittent HTTP 503 Code Failures
Troubleshoot Postgres Database
View Postgres Information for System Databases
System Recovery
PBS Service Recovery
Slurm Service Recovery
Beta Procedures for System Recovery
cani
Add A Blade To A Cabinet In SLS Using CANI
Add A Cabinet To SLS using CANI
multi-tenancy
Cray HNC Manager
Creating a Tenant
Example Workflow
Global Tenant Hooks
Modifying a Tenant
Multi-Tenancy Support
Removing a Tenant
Slurm Operator
TAPMS (Tenant and Partition Management System) Overview
Tenant Administrator Configuration
Multi-Tenancy Vault Overview
HPE Slingshot Network Operator
spire
Restore missing Spire metadata
Restore Spire Postgres without an Existing Backup
Spire Service Recovery
Troubleshoot Spire Failing to Start on NCNs
Update Spire Intermediate CA Certificate
Xname Validation
system configuration service
Configure BMC and Controller Parameters with SCSD
Manage Parameters with the scsd Service
Set BMC Credentials Using SAT
System Configuration Service
Cray System Management Install
SHCD HMN Tab/HMN Connections Rules
Ceph CSI Troubleshooting
Collect MAC Addresses for NCNs
Troubleshooting Installation Problems
Collecting the BMC MAC Addresses
PXE Boot Troubleshooting
Collecting NCN MAC Addresses
Troubleshooting Unused Drives on Storage Nodes
Configure Administrative Access
Utility Storage Installation Troubleshooting
Pre-Installation
Configure Management Network
Prepare Compute Nodes
Create Application Node Config YAML
Prepare site init
Create Cabinets YAML
Re-Installation
Create HMN Connections JSON File
Create NCN Metadata CSV
Create Switch Metadata CSV
Create System Configuration Using Cluster Discovery Service
Create System Configuration Using SHCD
CSM Services Install Fails Because of Missing Secret
Deploy Final NCN
Deploy Management Nodes
Install CSM Services
livecd
Accessing LiveCD USB Device After Reboot
Boot LiveCD RemoteISO
Boot LiveCD USB
Reinstall LiveCD
Reset root Password on a LiveCD USB
CSM Troubleshooting Information
Weave Container Network Interface Troubleshooting
Manual SSH Key Setting Process
Barebones Image Boot Test
Debugging With HMS PProf Images
DHCP Troubleshooting
DNS Troubleshooting
Error Rolling Back Service Chart With etcd
Running HMS CT Tests Manually
PXE Booting Runbook
Image projection inconsistent across nodes
Incrementally Configuring Images
Interpreting HMS Health Check Results
known issues
BOS Operator Pods OOMKilled
BOS Sessions Stuck Pending
CFS Component With Zero-Length ID
Gigabyte BMC Missing Redfish Data
HMS Resource Leaks
Istio-Proxy failing with too many open files
Keycloak Error "Cannot read properties" in Web UI
Nexus Fails Authentication with Keycloak Users
Boot content projection services can fail if an image does not have an etag
SLS Not Working During Node Rebuild
VCS Password With Illegal Characters
Known Issue admin-client-auth Not Found
Antero node NID allocation
Barebones Boot Test Failure
Known Issue Ceph OSD latency
Ceph OSD pool fails to create
cfs-api pods in CLBO state during CSM install
SAT/HSM/CAPMC/PCS Component Power State Mismatch
Console SSH Key Permissions
cray-console-node pods in CrashLoopBackOff
Cray CLI 403 Forbidden Errors
HMS Discovery Job Not Creating RedfishEndpoints In Hardware State Manager
Boot content projection services can fail if iscsid and multipathd services are not enabled on compute nodes and UANs
Flags Set For Nodes In HSM
Goss Test Fails with Connection Refused
Helm Chart Deploy Timeouts
HPE iLO dropping event subscriptions and not properly transitioning power state in CSM software
Known Issue IMS image creation failure
Known Issue IMS Image Delete Loses arch
Known Issue IMS Images Orphaned in S3
Known Issue IMS remote node build failure
Known issues with NCN health checks
IUF CLI reports false error that management-nodes-rollout failed
IUF does not run the next stage for an activity
IUF waits for 10 minutes to get workflow status before executing
cray-kafka-operator chart failing to upgrade
Known issue kubectl logs -f returns no space left on device
Mellanox lacp-individual Limitations
Missing binaries in aarch64 Images
NCN Health Checks Fail With cray-spire
Known issues with NCN resource checks
iSCSI SBPS projection over NMN will fail if DNS "A" (address) records for NMN are not created
HPE Cray EX255a Boot Issue with Console Parameter
Transaction Size Limitation for PCS and CAPMC
PostgreSQL Database is in Recovery
Product Catalog Upgrade Error
Remove User Access Service
Software Management Services health checks
Spire database connection pool configuration in an air-gapped environment
Spire Database Cluster DNS Lookup Failure
Spire Server Pods stuck in Pod Initializing
Enable Spire XName Validation Error
SSL Certificate Validation Issues
Storage node cloud-init fails with 'Timed out waiting for device' error
Test Failures Due To No Discovered Compute Nodes In HSM
Known Issue Velero Version Mismatch
Wait for unbound or cray-dns-unbound-manager hangs
kubernetes
Kubernetes kube-apiserver Failing
Kubernetes Log File Locations
Kubernetes pods failing to mount PVCs
Kubernetes Troubleshooting Information
Troubleshoot Kubernetes Master or Worker node in NotReady state
Troubleshoot Kubernetes Pods Not Starting
Troubleshoot Liveliness or Readiness Probe Failures
Troubleshoot Unresponsive kubectl Commands
Glossary
Introduction to CSM Installation
CSM Overview
View CSM documentation
Deprecated Features
CAPMC Deprecation Notice
Documentation Conventions
templates
Templates
Non-Compute Nodes
Certificate Authority
NCN BIOS
NCN Boot Workflow
NCN Firmware
NCN Images
Kernel Dumps
NCN Kernel
NCN Mounts and Filesystems
NCN Networking
NCN Operating System Releases
NCN Plan of Record
REST API Documentation
Boot Orchestration Service v2
Boot Script Service v1
Cray Advanced Platform Monitoring and Control (CAPMC) v3
Configuration Framework Service v1
Firmware Action Service v1
Heartbeat Tracker Service v1
HMS Notification Fanout Daemon v1
Image Management Service v3
NCN Lifecycle Service v1
Power Control Service (PCS) v1
System Configuration Service v1
System Layout Service v2
Hardware State Manager API v2
Cray STS Token Generator v1
TAPMS Tenant Status API v1
Update CSM Product Stream
Upgrade CSM
CSM 1.6.2 Patch Installation Instructions
Prepare for Upgrade to Next CSM Major Version
Resource Materials
k8s
Worker-Specific Manual Steps
storage
CEPHADM
CSM 1.5 to 1.6 Upgrade Process
Upgrade only CSM through IUF
Validate CSM Health During a CSM Upgrade
Manual NCN upgrade
scripts
sls
SLS Updates Expert mode
Upgrade SLS Offline from CSM 1.0.x to CSM 1.2
sls updater.py Technical Details
sls utils Library
upgrade
Upgrade Automation
workflows
iuf
operations
Argo Templates
Argo Templates
Cray System Management Documentation
>
CSM Troubleshooting Information
> known issues
known issues
Topics:
BOS Operator Pods OOMKilled
BOS Sessions Stuck Pending
CFS Component With Zero-Length ID
Gigabyte BMC Missing Redfish Data
HMS Resource Leaks
Istio-Proxy failing with too many open files
Keycloak Error “Cannot read properties” in Web UI
Nexus Fails Authentication with Keycloak Users
Boot content projection services can fail if an image does not have an etag
SLS Not Working During Node Rebuild
VCS Password With Illegal Characters
Known Issue admin-client-auth Not Found
Antero node NID allocation
Barebones Boot Test Failure
Known Issue Ceph OSD latency
Ceph OSD pool fails to create
cfs-api pods in CLBO state during CSM install
SAT/HSM/CAPMC/PCS Component Power State Mismatch
Console SSH Key Permissions
cray-console-node pods in CrashLoopBackOff
Cray CLI 403 Forbidden Errors
HMS Discovery Job Not Creating RedfishEndpoints In Hardware State Manager
Boot content projection services can fail if iscsid and multipathd services are not enabled on compute nodes and UANs
Flags Set For Nodes In HSM
Goss Test Fails with Connection Refused
Helm Chart Deploy Timeouts
HPE iLO dropping event subscriptions and not properly transitioning power state in CSM software
Known Issue IMS image creation failure
Known Issue IMS Image Delete Loses arch
Known Issue IMS Images Orphaned in S3
Known Issue IMS remote node build failure
Known issues with NCN health checks
IUF CLI reports false error that management-nodes-rollout failed
IUF does not run the next stage for an activity
IUF waits for 10 minutes to get workflow status before executing
cray-kafka-operator chart failing to upgrade
Known issue kubectl logs -f returns no space left on device
Mellanox lacp-individual Limitations
Missing binaries in aarch64 Images
NCN Health Checks Fail With cray-spire
Known issues with NCN resource checks
iSCSI SBPS projection over NMN will fail if DNS “A” (address) records for NMN are not created
HPE Cray EX255a Boot Issue with Console Parameter
Transaction Size Limitation for PCS and CAPMC
PostgreSQL Database is in Recovery
Product Catalog Upgrade Error
Remove User Access Service
Software Management Services health checks
Spire database connection pool configuration in an air-gapped environment
Spire Database Cluster DNS Lookup Failure
Spire Server Pods stuck in Pod Initializing
Enable Spire XName Validation Error
SSL Certificate Validation Issues
Storage node cloud-init fails with ‘Timed out waiting for device’ error
Test Failures Due To No Discovered Compute Nodes In HSM
Known Issue Velero Version Mismatch
Wait for unbound or cray-dns-unbound-manager hangs