Cray System Management
v
1.7
1.6
1.5
1.4
1.3
1.2
1.0
0.9
Cray System Management (CSM) - Release Notes
Cray System Management (CSM) 1.7.1 Release Notes
Cray System Management (CSM) 1.7.1-patch.1 Release Notes
Cray System Management (CSM) 1.7.1-patch.2 Release Notes
Cray System Management (CSM) Administration Guide
System Admin Toolkit (SAT)
About SAT
Introduction to SAT
SAT Command Overview
SAT Dependencies
SAT in CSM
SAT Uninstall and Downgrade
SAT on an External System
SAT Usage
Change the BOS Version
Change the CFS Version
Configure API Gateway Retries in SAT
SAT Bootprep
SAT and IUF
SAT Configuration
Authenticate SAT Commands
Configure Multi-tenancy (Optional)
Generate SAT S3 Credentials
Set System Revision Information
Create a Backup of HMS Items for Reinstall
Slingshot Fabric Manager on baremetal
Configure FM (Fabric Manager) On Baremetal
Redeploy Fabric Manager Nodes
iSCSI SBPS (Scalable Boot Content Projection Service)
Best Practices To Follow To Avoid Issues With iSCSI SBPS
Managing Selective Node Personalization
iSCSI SBPS Metrics
iSCSI SBPS Verification
Steps To Follow After The Worker Node Rollout During CSM Upgrade
Rack Resiliency (RR)
Rack Resiliency ConfigMaps
Critical Services
Enabling Rack Resiliency During Install or Upgrade
Enabling Rack Resiliency on a Running System
Kyverno Policy
Manage Critical Services
Resiliency Monitoring Service (RMS)
Setup of Rack Resiliency
Troubleshooting Rack Resiliency
Zones
cray-rrs Deployment
Component Names (xnames)
Restore HSM
Validate CSM Health
Boot Orchestration Service (BOS)
BOS API
Boot Orchestration
BOS Workflows
Compute Node Boot Issue Symptom Node Console or Logs Indicate That the Server Response has Timed Out
Boot Issue Symptom Node HSN Interface Does Not Appear or Show Detected Links Detected
Boot UANs
BOS Commands Cheat Sheet
Component Status
BOS Components
Compute Node Boot Issue Symptom Duplicate Address Warnings and Declined DHCP Offers in Logs
Compute Node Boot Issue Symptom Message About Invalid EEPROM Checksum in Node Console or Log
Compute Node Boot Issue Symptom Node Is Not Able to Download the Required Artifacts
Compute Node Boot Sequence
Create a Session Template to Boot Compute Nodes with Scalable Boot Projection Service (SBPS)
Customize iPXE Binary Names
BOS Database
Determine Which BOS Session Booted a Node
Edit the iPXE Embedded Boot Script
Exporting and Importing BOS Data
Exporting and Importing BSS Data
Healthy Compute Node Boot Process
Kernel Boot Parameters
Limit the Scope of a BOS Session
Log File Locations and Ports Used in Compute Node Boot Troubleshooting
Manage a BOS Session
Manage a Session Template
Multi-tenancy with BOS
Node Boot Root Cause Analysis
BOS Operators
BOS Options
Redeploy the iPXE and TFTP Services
BOS Reporter
Rolling Upgrades Using BOS
BOS Session Templates
BOS Sessions
Simple BOS Session Flow
Staging Changes with BOS
Tools for Resolving Compute Node Boot Issues
Troubleshoot Compute Node Boot Issues Related to Dynamic Host Configuration Protocol (DHCP)
Troubleshoot Compute Node Boot Issues Related to Trivial File Transfer Protocol (TFTP)
Troubleshoot Compute Node Boot Issues Related to Unified Extensible Firmware Interface (UEFI)
Troubleshoot Compute Node Boot Issues Related to the Boot Script Service (BSS)
Troubleshoot Compute Node Boot Issues Using Kubernetes
Troubleshoot UAN Boot Issues
Upload Node Boot Information to Boot Script Service (BSS)
View the Status of a BOS Session
Configure the Cray Command Line Interface (cray CLI)
kubernetes
About Kubernetes Taints and Labels
Kubernetes Encryption
About Postgres
About etcd
About kubectl
Backups for Etcd Clusters Running in Kubernetes
Kubernetes and Bare Metal EtcD Certificate Renewal
Check for and Clear etcd Cluster Alarms
Check the Health of etcd Clusters
Clear Space in an etcd Cluster Database
Configure kubectl Credentials to Access the Kubernetes APIs
containerd
Create a Manual Backup of Bare-Metal etcd Cluster
Create a Manual Backup of a Healthy etcd Cluster
Determine if Pods are Hitting Resource Limits
Disaster Recovery for Postgres
Fix Failed to start etcd on Master NCN
Increase Kafka Pod Resource Limits
Increase the PVC size in an etcd Cluster Database
Increase Pod Resource Limits
Kubernetes
Kubernetes Networking
Kubernetes Storage
Kyverno policy management
Pod Resource Limits
Rebuild Unhealthy etcd Clusters
Recover from Postgres WAL Event
Repopulate Data in etcd Clusters When Rebuilding Them
Report the Endpoint Status for etcd Clusters
Restore Bare-Metal etcd Clusters from an S3 Snapshot
Restore Postgres
Restore an etcd Cluster from a Backup
Retrieve Cluster Health Information Using Kubernetes
TDS Lower CPU Requests
Troubleshoot Intermittent HTTP 503 Code Failures
Troubleshoot Postgres Database
View Postgres Information for System Databases
spire
Enable TPM node attestation with Spire
Restore missing Spire metadata
Restore Spire Postgres without an Existing Backup
Spire Service Recovery
Troubleshoot Spire Failing to Start on NCNs
Update Spire Intermediate CA Certificate
Xname Validation
CSM product management
Security Hardening
Change Passwords and Credentials
Configure CSM Packages with CFS
Configure Keycloak Account
Configure the root Password and SSH Keys in Vault
Post-Install Customizations
Redeploying a Chart
Remove Artifacts from Product Installations
Set Up Passwordless SSH
Validate Signed RPMs
power management
Cray Advanced Platform Monitoring and Control (CAPMC)
Node Card Power Management
Power Control Service (PCS)
Power Off Compute Cabinets
Power Off Management Cabinets
Power Off Storage Cabinets
Power Off the External Lustre File System
Power On Compute Cabinets
Power On and Boot Managed Nodes
Power On and Start the Management Kubernetes Cluster
Power On the External Lustre File System
Prepare the System for Power Off
Recover from a Liquid Cooled Cabinet EPO Event
Save Management Network Switch Configuration Settings
Set the Turbo Boost Limit
Shut Down and Power Off Managed Nodes
Shut Down and Power Off the Management Kubernetes Cluster
System Power Off Procedures
System Power On Procedures
User Access to Compute Node Power Data
Power Management
firmware
FASUpdate Script
FAS Admin Procedures
FAS CLI
Cleaning Up FAS Database
FAS Filters
Backup and Restoring FAS Images
Updating Foxconn Paradise Nodes with FAS
FAS Recipes
Update iLO 5 Firmware Above v2.78
FAS Recipes and Procedures
Firmware Upgrade Using SPP on HPE ProLiant Servers
Update Firmware with FAS
Updating BMC Firmware and BIOS for ncn-m001
Updating BMC Firmware and BIOS for NCNs Without FAS
Upload BMC Recovery Firmware into TFTP Server
iuf
Install and Upgrade Framework
examples
iuf abort Examples
iuf activity Examples
iuf list-activities Examples
iuf list-stages Examples
iuf restart Examples
iuf resume Examples
iuf run Examples
iuf workflow Examples
stages
deliver-product
deploy-product
managed-nodes-rollout
management-nodes-rollout
post-install-check
post-install-service-check
pre-install-check
prepare-images
process-media
update-cfs-config
update-vcs-config
workflows
Populate Admin Directory with Files Defining Site Preferences
Backup
Migrating Kubernetes CNI from Weave to Cilium
Configuration
Configuration of the HPE Slingshot Fabric Manager
Deploy Product
Image Preparation
Install or Upgrade Additional Products with IUF
Managed Rollout
Management Rollout
Prepare for the Install or Upgrade
Product Delivery
Perform HPE Slingshot Switch and Management Network Switch Firmware Updates
Upgrade CSM and Additional Products with IUF
Validate Deployment
multi-tenancy
Cray HNC Manager
Creating a Tenant
Example Workflow
Global Tenant Hooks
Modifying a Tenant
Multi-Tenancy Support
Removing a Tenant
Slurm Operator
TAPMS (Tenant and Partition Management System)
Tenant Administrator Configuration
Multi-Tenancy Vault Overview
HPE Slingshot Network Operator
argo
Using Argo Workflows
Using the Argo UI
artifact management
Artifact Management
Generate Temporary S3 Credentials
Manage Artifacts with the Cray CLI
Use S3 Libraries and Clients
cani
Add A Blade To A Cabinet In SLS Using CANI
Add A Cabinet To SLS Using CANI
image management
Build a New UAN Image Using the Default Recipe
Build an Image Using IMS REST Service
Change IMS Job Log Level
Configure IMS to Use DKMS
Configure IMS to Validate RPMs
Configure a Remote Build Node
Convert TGZ Archives to SquashFS Images
Customize an Image Root Using IMS
Delete or Recover Deleted IMS Content
Exporting and Importing IMS Data
Image Job Performance
Image Management
Image Management Workflows
Import an External Image to IMS
Import an NCN Image to IMS
Troubleshoot Issues with Large Images
Troubleshoot Remote Build Node
Troubleshoot Interactions with zypper
Upload and Register an Image Recipe
Working With aarch64 Images
utility storage
Adding a Ceph Node to the Ceph Cluster
Add Ceph OSDs
Adjust Ceph Pool Quotas
Alternate Storage Pools
Ceph Daemon Memory Profiling
Ceph Deep Scrubs
Ceph Health States
Ceph Orchestrator Usage
Ceph Service Check Script Usage
Ceph Storage Types
ceph-upgrade-tool.py Usage
Cephadm Reference Material
Collect Information about the Ceph Cluster
Dump Ceph Crash Data
Identify Ceph Latency Issues
Manage Ceph Services
Move Unmanaged Ceph OSDs
Shrink the Ceph Cluster
Shrink Ceph OSDs
Troubleshoot Ceph-Mon Processes Stopping and Exceeding Max Restarts
Troubleshoot Ceph MDS Client Connectivity Issues
Troubleshooting Ceph MDS Reporting Slow Requests and Failure on Client
Troubleshoot Ceph New RGW Deployment Failing
Troubleshoot Ceph OSDs Not Being Created on Disks
Troubleshoot Ceph OSDs Reporting Full
Troubleshoot Ceph Services Not Starting After a Server Crash
Troubleshoot Failure to Get Ceph Health
Troubleshoot HEALTH ERR Module devicehealth has failed table Device already exists
Troubleshoot Insufficient Standby MDS Daemons Available
Troubleshoot Large Object Map Objects in Ceph Health
Troubleshoot Pods Multi-Attach Error
Fixing incorrect number of PG Issues
Troubleshoot if RGW Health Check Fails
Troubleshoot S3FS Cache Cleanup
Troubleshoot S3 Filesystem Mount Issues
Troubleshoot System Clock Skew
Troubleshoot a Down OSD
Troubleshoot an Unresponsive Rados-Gateway (radosgw) S3 Endpoint
Troubleshoot Ceph image with tag <none>
Utility Storage
Update ceph node-exporter config to monitor SNMP counters
system layout service
Add Liquid-Cooled Cabinets to SLS
Add UAN CAN IP Addresses to SLS
Add an alias to a service
Create a Backup of the SLS Postgres Database
Dump SLS Information
Load SLS Database with Dump File
Restore SLS Postgres Database from Backup
Restore SLS Postgres without an Existing Backup
System Layout Service (SLS)
Update SLS with UAN Aliases
observability
Install and Upgrade Observability Framework
security and authentication
API Authorization
Access the Keycloak User Management UI
Add LDAP User Federation
Add Root Service Account for Gigabyte Controllers
Audit Logs
Authenticate an Account with the Command Line
Backup and Restore Vault Clusters
Certificate Types
Change Air-Cooled Node BMC Credentials Using SAT
Change Credentials on ServerTech PDUs
Change Cray EX Liquid-Cooled Cabinet Global Default Password
Change the Keycloak Token Lifetime
Set NCN Image Root Password, SSH Keys, and Timezone
Set NCN Image Root Password, SSH Keys, and Timezone on PIT Node
Change Root Passwords for Compute Nodes
Change the Keycloak Admin Password
Change the LDAP Server IP Address for Existing LDAP Server Content
Change the LDAP Server IP Address for New LDAP Server Content
Configure Keycloak for LDAP/AD authentication
Configure root user on HPE iLO BMCs
Configure the RSA Plugin in Keycloak
Create Internal Groups in the Keycloak Shasta Realm
Create Internal User Accounts in the Keycloak Shasta Realm
Create a Backup of the Keycloak Postgres Database
Create a Service Account in Keycloak
Default Keycloak Realms, Accounts, and Clients
Delete Internal User Accounts in the Keycloak Shasta Realm
Get a Long-Lived Token for a Service Account
HashiCorp Vault
Keycloak Operations
Keycloak Service Recovery
Keycloak User Localization
Keycloak User Management with kcadm.sh
Make HTTPS Requests from Sources Outside the Management Kubernetes Cluster
Manage Sealed Secrets
Manage System Passwords
PKI Certificate Authority (CA)
PKI Services
Preserve Username Capitalization for Users Exported from Keycloak
Provisioning a Liquid-Cooled EX Cabinet CEC with Default Credentials
Public Key Infrastructure (PKI)
Recovering from Mismatched BMC Credentials
Remove Internal Groups from the Keycloak Shasta Realm
Remove the Email Mapper from the LDAP User Federation
Remove the LDAP User Federation from Keycloak
Re-Sync Keycloak Users to Compute Nodes
Retrieve an Authentication Token
Retrieve the Client Secret for Service Accounts
SOPS Introduction
Update NCN User SSH Keys
System Security and Authentication
Transport Layer Security (TLS) for Ingress Services
Troubleshoot Common Vault Cluster Issues
Troubleshoot Kyverno configuration manually
Update Default Air-Cooled BMC and Leaf-BMC Switch SNMP Credentials
Update Default ServerTech PDU Credentials used by the Redfish Translation Service (RTS)
Set NCN User Passwords
Update Root Secrets In Vault
Updating the Liquid-Cooled EX Cabinet CEC with Default Credentials after a CEC Password Change
Vault Service Recovery
Keycloak IPv6 Support
System Recovery
PBS Service Recovery
Slurm Service Recovery
Beta Procedures for System Recovery
configuration management
Accessing sat bootprep Files
Adding Additional Inventory
Ansible Execution Environments
Ansible Log Collection
Automatic Configuration Management
Automatic Session Deletion with session ttl
Backup and Restore VCS Data
CFS Batcher
CFS Commands Cheat Sheet
CFS Components
CFS Configurations
CFS Flow
CFS Global Options
CFS Hardware Synchronization Agent
CFS Key Management and Ansible Permission Denied Errors
CFS Operator
CFS Session Inventory
CFS Sessions
CFS Sources
CFS State Reporter
Change the Ansible Verbosity
Configuration Management
Configure Ansible
Configure HSN NIC Bonding
Create a Node Personalization CFS Session
Create an Image Customization CFS Session
Create and Populate a VCS Configuration Repository
Customize Configuration Values
Differences Between the V2 and V3 CFS APIs
Enable Ansible Profiling
Exporting and Importing CFS Data
Git Operations
Management Node Image Customization
Management Node Personalization
Managing Sensitive Tenant Information in VCS with SOPS
Paging CFS Records
Set Limits for a Configuration Session
Specifying Hosts and Groups
Target Ansible Tasks for Image Customization
Track the Status of a Session
Troubleshoot CFS Issues
Troubleshoot Failed CFS Sessions
Troubleshoot CFS Session Failing to Complete
Troubleshoot CFS Sessions Failing to Start
Update a CFS Configuration
Update the Privacy Settings for Gitea Configuration Content Repositories
VCS Administrative User
VCS Branching Strategy
Version Control Service (VCS)
View Configuration Session Logs
Write Ansible Code for CFS
hpe pdu
HPE PDU Admin Procedures
network
Management Network User Guide
Manual Switch Configuration
Fresh Install
Added Hardware
Generate Switch Configurations
Apply Custom Switch Configuration CSM 1.2
Apply Switch Configurations
CSM Automatic Network Utility
CANU Installation
Troubleshoot CANU Validation Errors
Use CANU to Verify, Generate, or Compare Switch Configurations
Generate Switch Configurations Including Custom Configurations
Initializing CANU
Introduction to CANU
Quick start guide to CANU
Uninstall CANU
Update CANU From CSM Release Tarball
Use CANU to Generate Full Network Configuration
Dell Installation and Configuration Guide
Access Control Lists (ACLs)
Address Resolution Protocol (ARP)
Backing Up A Switch Configuration
Domain Name System (DNS) Client
Domain Name
Hostname
Internet Group Multicast Protocol (IGMP)
Link Aggregation Group (LAG)
Link Layer Discovery Protocol (LLDP)
Configure Locator LED
Loopback Interface
Management Interface
Multiple Spanning Tree Protocol (MSTP)
Network Time Protocol (NTP) Client
Physical Interfaces
Configure QoS
Remote Logging
Reset Dell Switch Configuration
Configure SNMPv2c community
Dell SNMPv3 Users
Secure Shell (SSH)
Configure System Images
Perform an Upgrade on Dell Switches
Configure Virtual Local Access Networks (VLANs)
VLAN Interface
VLAN Trunking 802.1Q
Using canu-inventory with Ansible
Upgrade CANU
Procedure to check CHN
Collect Data
Configuration Management
Configuring SNMP in CSM
Mellanox Installation and Configuration Guide
Access Control Lists (ACLs)
Address Resolution Protocol (ARP)
Backing Up A Switch Configuration
Border Gateway Protocol (BGP) Basics
Cable Diagnostics
Check BGP and MetalLB
Check Current DHCP Leases
Check DHCP Lease is Getting Allocated
Check HSM
Check KEA DHCP Logs
Verify Computes/UANs/Application Nodes
Large Number of DHCP Declines During a Node Boot
Domain Name System (DNS) Client
Domain Name
Duplicate IP Addresses
Exec Banners
Hostname
Internet Group Multicast Protocol (IGMP)
IP Filter
Key Features Used in the Management Network Configuration
Link Aggregation Group (LAG)
Large Network
Link Layer Discovery Protocol (LLDP)
Loopback Interface
Management Interface
Management Network Configuration Example
Management Network Functions in Detail
Medium Network
Multi-Chassis Link Aggregation Group (MCLAG)
MLAG (Multi-Chassis LAG)
MLAG
Multiple Spanning Tree Protocol (MSTP)
Native VLAN
NCN tcpdump
NCNs on Install
Network Types – Naming and Segment Function
Network Traffic Pattern
Network Time Protocol (NTP) Client
Open Shortest Path First (OSPF) v2
Physical Interfaces
PIM-SM Bootstrap Router (BSR) and Rendezvous Point (RP)
Rebooting NCNs and PXE Fails
Remote Logging
Connect the Management Network to a Campus Network
Routed Interfaces
Scenario A Network Connection via Management Network
Scenario B Network Connection via High-Speed Network
Small Network
SNMPv2c Community
Mellanox SNMPv3 Users
Spine-Leaf Architecture
Spine-leaf Architecture
Why Are Spine-leaf Architectures Becoming More Popular?
Secure Shell (SSH)
MAC Address Table
Static Routing
Confirm the Status of the cray-dhcp-kea Pods
System Images
Test TFTP traffic (Aruba Only)
Typical Configuration of MLAG Link Connecting to NCN
Typical Configuration of MLAG Between Switches
Performing Upgrade On Mellanox Switches
Verify the Switches are Forwarding DHCP Traffic
Verify BGP
Verify the DHCP Traffic on the Worker Nodes
Verify Route to TFTP
Very Large (Exascale) Network
Virtual Local Access Networks (VLANs)
VLAN Interface
VLAN Trunking 802.1Q
Web User Interface (Web UI)
Aruba Installation and Configuration Guide
802.1X
Access Control Lists (ACLs)
Address Resolution Protocol (ARP)
Backing Up A Switch Configuration
Border Gateway Protocol (BGP) Basics
Bluetooth Capabilities
Cable Diagnostics
Check BGP and MetalLB
Check Current DHCP Leases
Check DHCP Lease is Getting Allocated
Check HSM
Check KEA DHCP Logs
Classifier Policies
Verify Computes/UANs/Application Nodes
Large Number of DHCP Declines During a Node Boot
Domain Name System (DNS) Client
Domain Name
Duplicate IP Addresses
Exec Banners
Hostname
Internet Group Multicast Protocol (IGMP)
Initial Prioritization
Introduction
Key Features Used in the Management Network Configuration
Link Aggregation Group (LAG)
Link Layer Discovery Protocol (LLDP)
Locator LED
Loopback Interface
MAC Authentication
Management Interface
Management Network Configuration Example
Management Network Functions in Detail
VSX ISL HA
VSX MCLAG Link HA
VSX Member Power Failure
VSX Split
Multi-Chassis Link Aggregation Group (MCLAG)
Message-Of-The-Day (MOTD)
Multicast Source Discovery Protocol (MSDP)
Multiple Spanning Tree Protocol (MSTP)
Native VLAN
NCN tcpdump
NCNs on Install
Network Types – Naming and Segment Function
Network Topologies
Network Traffic Pattern
Notices
Network Time Protocol (NTP) Client
Open Shortest Path First (OSPF) v2
Physical Interfaces
PIM-SM Bootstrap Router (BSR) and Rendezvous Point (RP)
Port Mirroring
Port Security
Queuing and Scheduling
RADIUS
Rebooting NCNs and PXE Fails
Redundant Power Supplies
Remote Logging
Connect the Management Network to a Campus Network
Routed Interfaces
Scenario A Network Connection via Management Network
Scenario B Network Connection via High-Speed Network
Simple Network Management Protocol (SNMP) Agent
SNMPv2c Community
SNMP Traps
Aruba SNMPv3 Users
Spine-Leaf Architecture
Spine-leaf Architecture
Secure Shell (SSH)
Static Routing
Confirm the Status of the cray-dhcp-kea Pods
TACACS
Test TFTP traffic (Aruba Only)
Typical Configuration of VSX
Typical Edge Port Configuration
Typical Configuration of MCLAG Link
Unidirectional Link Detection (UDLD)
Perform a VSX Upgrade on Aruba Switches
Verify the Switches are Forwarding DHCP Traffic
Verify BGP
Verify the DHCP Traffic on the Worker Nodes
Verify Route to TFTP
Virtual Local Access Networks (VLANs)
VLAN Interface
VLAN Trunking 802.1Q
Virtual Switching Framework (VSF) - 6300 Only
Virtual Switching Extension (VSX)
VSX Architecture
Switch Replacement in the VSX Cluster
VSX Sync
Web User Interface (Web UI)
Erase All zeroize
Edge switch cabling guide
Network Tests
Reinstall
Replace Switch
Save A Configuration
Prometheus SNMP Exporter
Transceivers and Cables
Example of the Connections Used in Shasta Management Network
Validate Cabling
Validate the SHCD
Validate Switch Configurations
Wipe Management Switch Configuration
Aruba splitting of QSFP+ and QSFP28 ports
Backup a Custom Configuration
BICAN Support Matrix - Shasta Customer Access Networks
BICAN switch configuration
Bifurcating the CAN - Feature Details
BICAN Summary
Bonded UAN Configuration
Cable Management Network Servers
firmware
Update Management Network Firmware
hardware
EX2500 Installation and Cabling
ARP Cache Tuning Guide
Access to System Management Services
Connect to Switch over USB-Serial Cable
Connect to the HPE Cray EX Environment
Create a CSM Configuration Upgrade Plan
Default IP Address Ranges
Gateway Testing
Node Management Network (NMN) Isolation
Network
customer accessible networks
IPv6 Configuration Guide
Connect to the CMN and CAN
Customer Access Networks
network
Enabling Customer High Speed Network Routing
Management Network Upgrade CSM 1.2 to 1.3
scripts
sls
sls utils Library
Customer Accessible Networks
CAN/CMN with Dual-Spine Configuration
Externally Exposed Services
Troubleshoot CMN Issues
BI-CAN Aruba/Arista Configuration
MetalLB Peering with Arista Edge Router
metallb bgp
Check BGP Status and Reset Sessions
MetalLB Configuration
MetalLB in BGP-Mode
Troubleshoot BGP not Accepting Routes from MetalLB
Troubleshoot Services without an Allocated IP Address
dhcp
DHCP boot file customization
DHCP
Troubleshoot DHCP Issues
dns
Domain Name Service (DNS) Overview
Manage the DNS Unbound Resolver
PowerDNS Configuration
PowerDNS Migration Guide
Troubleshoot Common DNS Issues
Troubleshoot PowerDNS
external dns
External DNS
External DNS Failing to Discover Services Workaround
External DNS CSI Input Values
Ingress Routing
Troubleshoot DNS Configuration Issues
Troubleshoot Connectivity to Services with External IP addresses
Update the cmn-external-dns value post-installation
system configuration service
Configure BMC and Controller Parameters with SCSD
Manage Parameters With The SCSD Service
Set BMC Credentials Using SAT
System Configuration Service
bare metal
Bare-Metal Steps
Fresh Install Setting NodeBMC and RouterBMC Redfish Credentials
conman
Access Compute Node Logs
Access Console Log Data Via the System Monitoring Framework (SMF)
Complete Reset of the Console Services
ConMan
Configure Log Rotation
Console Services Troubleshooting Guide
Disable ConMan After the System Software Installation
Establish a Serial Connection to NCNs
Log in to a Node Using ConMan
Manage Node Consoles
Troubleshoot ConMan Asking for Password on SSH Connection
Troubleshoot ConMan Blocking Access to a Node BMC
Troubleshoot ConMan Failing to Connect to a Console
Troubleshoot Console Node Pod Stuck in Terminating State
system management health
Access System Management Health Services
Cilium and Hubble Monitoring
Configure Prometheus Email Alert Notifications
Retrieve SMART data from ClusterStor E1000 nodes via Redfish Exporter
Grafana Dashboards by Component
grok-exporter pod status showing as ContainerStatusUnknown Error
prometheus-kafka-adapter errors during installation
Remove Kiali
SMA Victoria Metrics Monitoring
System Management Health
System Management Health Checks and Alerts
Troubleshoot Grafana Dashboard
Troubleshoot Prometheus Alerts
UAN Node Exporter
VictoriaMetrics
hardware state manager
Add a Switch to the HSM Database
Add an NCN to the HSM Database
Component Group Members
Component Groups and Partitions
Component Memberships
Component Partition Members
Create a Backup of the HSM Postgres Database
Backup/Restore HSM User Data (Locks, Groups, and Partitions)
HSM Roles and Subroles
Hardware Management Services (HMS) Locking API
Hardware State Manager (HSM)
Hardware State Manager (HSM) State and Flag Fields
Lock and Unlock Management Nodes
Manage Component Groups
Manage Component Partitions
Manage HSM Locks
Remove Duplicate Detected Events From the HSM Postgres Database
Restore Hardware State Manager (HSM) Postgres Database from Backup
Restore Hardware State Manager (HSM) Postgres without an Existing Backup
Set BMC Management Roles
hmcollector
Adjust HM Collector Ingress Replicas and Resource Limits
node management
Access and Update Settings for Replacement NCNs
Removing a Liquid-cooled blade from a System
Removing a Liquid-cooled blade from a System Using SAT
Removing a Standard rack node from a System
Replace a Compute Blade
Replace a Compute Blade Using SAT
Replace a Standard rack node from a System
Replacing Foxconn Username and Passwords in Vault
Add TLS Certificates to BMCs
Repurpose a Compute Node as a UAN
Add a Standard Rack Node
Reset Credentials on Redfish Devices
Add Additional Air-Cooled Cabinets to a System
S3FS Usage and Guidelines for Shasta
Add Additional Liquid-Cooled Cabinets to a System
Set Gigabyte Node BMC to Factory Defaults
Adding a Liquid-cooled Blade to a System
Swap a Compute Blade with a Different System
Adding a Liquid-cooled blade to a System Using SAT
Swap a Compute Blade with a Different System Using SAT
Build NCN Images Locally
Switch PXE Boot from Onboard NIC to PCIe
Change Java Security Settings
TLS Certificates for Redfish BMCs
Change Settings for HMS Collector Polling of Air-Cooled Nodes
Troubleshoot Interfaces with IP Address Issues
Check and Set the metal.no-wipe Setting on NCNs
Troubleshoot Issues with Redfish Endpoint Discovery
Check the BMC Failover Mode
Troubleshoot Loss of Console Connections and Logs on Gigabyte Nodes
Clear Space in Root File System on Worker Nodes
Update Compute Node Mellanox HSN NIC Firmware
Configuration of NCN Bonding
Update the Gigabyte Node BIOS Time
Configure NTP on NCNs
Update the HPE Node BIOS Time
Customize PCIe Hardware
Updating Cabinet Routes on Management NCNs
Customize PCIe Hardware
Use the Physical KVM
Defragment NID Numbering
Verify Node Removal
Disable Nodes
View BIOS Logs for Liquid-Cooled Nodes
Dump a Non-Compute Node
Manual Wipe Procedures
Enable Nodes
Clear Gigabyte CMOS
Enable Passwordless Connections to Liquid Cooled Node BMCs
Enable IPMI access on HPE iLO BMCs
Find Node Type and Manufacturer
Launch a Virtual KVM on Gigabyte Servers
Launch a Virtual KVM on Intel Servers
Move a Standard Rack Node
Move a Standard Rack Node (Same Rack/Same HSN Ports)
Move a liquid-cooled blade within a System
NCN Drive Identification
NCN NIC Replacement
NCN Network Troubleshooting
Node Management
Node Management Workflows
Reboot NCNs
Reboot NCNs with IUF
Reboot NCNs manually
Add Remove Replace NCNs
Add NCN Data
Alpha Framework to Add, Remove, Replace, or Move NCNs
Add Switch Configuration for NCN
Allocate NCN IP Addresses
Boot NCN
Collect NCN MAC Addresses
Redeploy Services Impacted by Adding or Permanently Removing Storage Nodes
Remove NCN Data
Remove NCN from Role
Remove Switch Configuration for NCN
Update Firmware
Update NCN BIOS TPM State
Validate Health
Validate Added NCN
Rebuild NCNs
Final Validation Steps
Identify Nodes and Update Metadata
Post Rebuild Storage Node Validation
Power Cycle and Rebuild Nodes
Prepare Storage Nodes
Re-Add a Storage Node to Ceph
Rebuild NCNs
Validate Boot Loader
resiliency
Recreate StatefulSet Pods on Another Node
Resilience of System Management Services
Resiliency
Resiliency Testing Procedure
Restore System Functionality if a Kubernetes Worker Node is Down
package repository management
Manage Repositories with Nexus
Nexus Configuration
Nexus Deployment
Nexus Export and Restore
Nexus Service Recovery
Nexus Space Cleanup
Package Repository Management
Package Repository Management with Nexus
Repair Blobstore
Repair Yum Repository Metadata
Restrict Admin Privileges in Nexus
Troubleshoot Nexus
Cray System Management Install
SHCD HMN Tab/HMN Connections Rules
Ceph CSI Troubleshooting
Collect MAC Addresses for NCNs
Troubleshooting Installation Problems
Collecting the BMC MAC Addresses
PXE Boot Troubleshooting
Collecting NCN MAC Addresses
Troubleshooting Unused Drives on Storage Nodes
Configure Administrative Access
Utility Storage Installation Troubleshooting
Pre-Installation
Configure Management Network
Prepare Compute Nodes
Create Application Node Config YAML
Prepare site-init
Create Cabinets YAML
Re-Installation
Create HMN Connections JSON File
Create NCN Metadata CSV
Create Switch Metadata CSV
Create System Configuration Using Cluster Discovery Service
Create System Configuration Using SHCD
CSM Services Install Fails Because of Missing Secret
Deploy Final NCN
Deploy Management Nodes
Install CSM Services
livecd
Accessing LiveCD USB Device After Reboot
Boot LiveCD RemoteISO
Boot LiveCD USB
Reinstall LiveCD
Reset root Password on a LiveCD USB
CSM Troubleshooting Information
Manual SSH Key Setting Process
Cilium Network Troubleshooting Runbook
CFS Sessions Race Condition Test
Barebones Image Boot Test
cmsdev Tests
Debugging With HMS pprof Images
DHCP Troubleshooting
PXE Booting Runbook
DNS Troubleshooting
Error Rolling Back Service Chart With etcd
Running HMS CT Tests Manually
Image projection inconsistent across nodes
Incrementally Configuring Images
Interpreting HMS Health Check Results
kubernetes
Kubernetes kube-apiserver Failing
Kubernetes Log File Locations
Kubernetes pods failing to mount PVCs
Kubernetes Troubleshooting Information
Troubleshoot Kubernetes Master or Worker node in NotReady state
Troubleshoot Kubernetes Pods Not Starting
Troubleshoot Liveliness or Readiness Probe Failures
Troubleshoot Unresponsive kubectl Commands
known issues
CFS Batcher Can Be Slow To See Updated CFS Options
CFS Batcher Creating Multiple Sessions For Same Batch
CFS Component State Update Does Not Preserve Layer Timestamps
CFS Component State Updates Allow Invalid Values
CFS Component With Zero-Length ID
CFS Operator Creating Multiple Jobs For Same Session
Gigabyte BMC Missing Redfish Data
IMS create job missing extended attributes support
Keycloak Error "Cannot read properties" in Web UI
Nexus Fails Authentication with Keycloak Users
Race Conditions in BOS and CFS
SLS Not Working During Node Rebuild
VCS Password With Illegal Characters
Known Issue admin-client-auth Not Found
Antero node NID allocation
Known Issue Ceph OSD latency
cfs-api Pods in CLBO State During CSM Install
CFS Session for Image Customization on Remote Node Status Stuck at running
Cilium Migration Failure Due to Missing BSS Global Metadata Parameter
SAT/HSM/PCS Component Power State Mismatch
cray-vault-operator Chart Upgrade Error
Cray CLI 403 Forbidden Errors
HMS Discovery Job Not Creating RedfishEndpoints In Hardware State Manager
etcd Pods in CLBO State
Flags Set For Nodes In HSM
Helm Chart Deploy Timeouts
HPE iLO dropping event subscriptions and not properly transitioning power state in CSM software
IMS Image Customization Job Status Stuck at waiting on user
Known Issue IMS Images Orphaned in S3
iSCSI SBPS systemd Service (sbps-marshal) May Fail During Upgrade
Known Issues With NCN Health Checks
Troubleshooting guide for IUF management-nodes-rollout to worker nodes hangs on final worker node with DVS related errors
IUF Unable To Run Next Stage
IUF waits for 10 minutes to get workflow status before executing
Keycloak Hung During Prerequisites Rollout
Node Unable to Join Kubernetes Cluster During NCN Rebuild
kubectl TLS Handshake Timeout Caused by HAProxy Connection Exhaustion
Kubelet Memory Pressure False Positive
Cloud init loops for master node unknown field udpIdleTimeout
Missing binaries in aarch64 Images
Known Issues With NCN Resource Checks
OAuth2 Proxy TLS Handshake Error
HPE Cray EX255a Boot Issue with Console Parameter
PostgreSQL Cluster Upgrades Failing
PostgreSQL System ID Mismatch
Worker nodes may hang or unresponsive post rootfs/PE image deletion
Software Management Services Health Check - Known Issues
Spire database connection pool configuration in an air-gapped environment
Spire Database Cluster DNS Lookup Failure
Spire Server Pods stuck in Pod Initializing
SSL Certificate Validation Issues
Test Failures Due To No Discovered Compute Nodes In HSM
While upgrading CSM having 1.6.x to 1.7.0, Kubernetes worker nodes may become unresponsive and dmesg may be flooded with iSCSI errors
Glossary
Introduction to CSM Installation
View CSM Documentation
csi Tool Changes
Deprecated Features
CAPMC Deprecation Notice
CSM Overview
Documentation Conventions
templates
Templates
Non-Compute Nodes
Certificate Authority
NCN BIOS
NCN Boot Workflow
NCN Firmware
NCN Images
Kernel Dumps
NCN Kernel
NCN Mounts and Filesystems
NCN Networking
NCN Operating System Releases
NCN Plan of Record
REST API Documentation
Boot Orchestration Service v2
Boot Script Service v1
Cray Advanced Platform Monitoring and Control (CAPMC) v3
Configuration Framework Service v1
Console Service v1
Firmware Action Service v1
Heartbeat Tracker Service v1
HMS Notification Fanout Daemon v1
Image Management Service v3
NCN Lifecycle Service v1
Power Control Service (PCS) v1
Rack Resiliency Service v1
System Configuration Service v1
System Layout Service v2
Hardware State Manager API v2
Cray STS Token Generator v1
TAPMS Tenant Status API v1
Update CSM Product Stream
Upgrade CSM
Prepare for Upgrade to Next CSM Major Version
Resource Materials
k8s
Worker-Specific Manual Steps
storage
CEPHADM
CSM 1.6 to 1.7 Upgrade Process
Upgrade Only CSM Through IUF
Validate CSM Health During a CSM Upgrade
Manual NCN Upgrade
scripts
sls
SLS Updates Expert Mode
Upgrade SLS Offline from CSM 1.0.x to CSM 1.2
sls updater.py Technical Details
sls utils Library
upgrade
Upgrade Automation
workflows
iuf
operations
Argo Templates
Argo Templates
Cray System Management Documentation
>
CSM Troubleshooting Information
> known issues
known issues
Topics:
CFS Batcher Can Be Slow To See Updated CFS Options
CFS Batcher Creating Multiple Sessions For Same Batch
CFS Component State Update Does Not Preserve Layer Timestamps
CFS Component State Updates Allow Invalid Values
CFS Component With Zero-Length ID
CFS Operator Creating Multiple Jobs For Same Session
Gigabyte BMC Missing Redfish Data
IMS create job missing extended attributes support
Keycloak Error “Cannot read properties” in Web UI
Nexus Fails Authentication with Keycloak Users
Race Conditions in BOS and CFS
SLS Not Working During Node Rebuild
VCS Password With Illegal Characters
Known Issue admin-client-auth Not Found
Antero node NID allocation
Known Issue Ceph OSD latency
cfs-api Pods in CLBO State During CSM Install
CFS Session for Image Customization on Remote Node Status Stuck at running
Cilium Migration Failure Due to Missing BSS Global Metadata Parameter
SAT/HSM/PCS Component Power State Mismatch
cray-vault-operator Chart Upgrade Error
Cray CLI 403 Forbidden Errors
HMS Discovery Job Not Creating RedfishEndpoints In Hardware State Manager
etcd Pods in CLBO State
Flags Set For Nodes In HSM
Helm Chart Deploy Timeouts
HPE iLO dropping event subscriptions and not properly transitioning power state in CSM software
IMS Image Customization Job Status Stuck at waiting on user
Known Issue IMS Images Orphaned in S3
iSCSI SBPS systemd Service (sbps-marshal) May Fail During Upgrade
Known Issues With NCN Health Checks
Troubleshooting guide for IUF management-nodes-rollout to worker nodes hangs on final worker node with DVS related errors
IUF Unable To Run Next Stage
IUF waits for 10 minutes to get workflow status before executing
Keycloak Hung During Prerequisites Rollout
Node Unable to Join Kubernetes Cluster During NCN Rebuild
kubectl TLS Handshake Timeout Caused by HAProxy Connection Exhaustion
Kubelet Memory Pressure False Positive
Cloud init loops for master node unknown field udpIdleTimeout
Missing binaries in aarch64 Images
Known Issues With NCN Resource Checks
OAuth2 Proxy TLS Handshake Error
HPE Cray EX255a Boot Issue with Console Parameter
PostgreSQL Cluster Upgrades Failing
PostgreSQL System ID Mismatch
Worker nodes may hang or unresponsive post rootfs/PE image deletion
Software Management Services Health Check - Known Issues
Spire database connection pool configuration in an air-gapped environment
Spire Database Cluster DNS Lookup Failure
Spire Server Pods stuck in Pod Initializing
SSL Certificate Validation Issues
Test Failures Due To No Discovered Compute Nodes In HSM
While upgrading CSM having 1.6.x to 1.7.0, Kubernetes worker nodes may become unresponsive and dmesg may be flooded with iSCSI errors