HPE Cray EX systems can have network switches in many roles: spine switches, leaf switches, leaf-bmc switches, and
Coolant Distribution Unit (CDU) switches.
Newer systems have HPE Aruba switches, while older systems have Dell and Mellanox switches. Switch IP addresses are generated by
Cray Site Init (CSI).
Documentation for the Management Network can be found in the HPE Cray EX Management Network Installation and Configuration Guide.
The configuration for these switches will be generated from the CSM Automatic Network Utility (CANU). For more details, see the CANU documentation.
It is assumed that the administrator configuring the Management Network has a basic understanding of networking protocols (STP, VLAN, OSPF, LAG/MLAG, BGP, etc.), and knows how to configure network equipment. It is also assumed that they understand and know how to read a Shasta Cabling Diagram (SHCD) file.
Before configuring/reconfiguring any switches, make sure to get the current running configuration and save that in case the configuration must be reverted.
All switch configuration should be done with a console or out of band connection.
(sw#) Save the output of the following by dumping the text configuration of the switch:
show running-configuration
The management network switches should be configured in this order: spine, leaf (if present), CDU (if present), and leaf-bmc.
Only systems with liquid-cooled cabinets will have the CDU switches. Only systems with many nodes in air-cooled cabinets
will have leaf switches.
The management network switches should be configured in this order: spine, leaf (if present), CDU (if present), and leaf-bmc.
Only systems with liquid-cooled cabinets will have the CDU switches. Only systems with many nodes in air-cooled
cabinets will have leaf switches.
On a typical system, the Mellanox switches are spine switches and the Dell switches are used for leaf, CDU, and leaf-bmc.
Currently CANU does not automatically create site connections (LAGs/uplink interfaces or default routes).
Hence, administrators need to manually configure the uplinks for site connections as well as default routes.
Example configuration:
NOTE These are very simplistic examples and depending on the install scenario, administrators may need to also configure LAGs, etc.
sw#
enable
conf term
interface ethernet 1/16 no switchport force
interface ethernet 1/16 speed 10G force
interface ethernet 1/16 vrf forwarding Customer
interface ethernet 1/16 ip address 10.102.255.10/30 primary
ip route vrf Customer 0.0.0.0/0 10.102.3.3
ip route vrf Customer 0.0.0.0/0 10.102.255.9
sw#
config
interface 1/1/16
vrf attach Customer
ip address 10.102.255.10/30
no shutdown
exit
ip route 0.0.0.0/0 10.102.3.3 vrf Customer
ip route 0.0.0.0/0 0.102.255.9 vrf Customer
After completing this procedure, collect MAC addresses for the management nodes using the PIT node and the management network switches configured in this procedure.