This page describes how Mellanox spine switches are configured.
Depending on the size of the HPE Cray EX system, the spine switches serve different purposes. On Test and Development Systems (TDS), the NCNs plug directly into the spine switches. On larger systems with aggregation switches, the spine switches provide connection between the aggregation switches.
Here is an example of spine switch data in an excerpt from the SHCD. The ISL ports are port 32 on both spine switches.
| Source | Source Label Info | Destination Label Info | Destination | Description |
|---|---|---|---|---|
sw-100g01 |
x3105u40-j32 |
x3105u41-j32 |
sw-100g02 |
100g-1m-DAC |
Cray Site Init (CSI) generates the IP addresses used by the system; below are examples only. The VLAN information is located in the network YAML files.
Verify the spine switches have VLAN interfaces in NMN, HMN, and CAN networks.
/var/www/ephemeral/prep/${SYSTEM_NAME}/networks/NMN.yaml - ip_address: 10.252.0.2
name: sw-spine-001
comment: x3000c0h12s1
aliases: []
- ip_address: 10.252.0.3
name: sw-spine-002
comment: x3000c0h13s1
aliases: []
name: network_hardware
net-name: NMN
vlan_id: 2
comment: ""
gateway: 10.252.0.1
/var/www/ephemeral/prep/${SYSTEM_NAME}/networks/HMN.yaml - ip_address: 10.254.0.2
name: sw-spine-001
comment: x3000c0h12s1
aliases: []
- ip_address: 10.254.0.3
name: sw-spine-002
comment: x3000c0h13s1
aliases: []
name: network_hardware
net-name: HMN
vlan_id: 4
comment: ""
gateway: 10.254.0.1
/var/www/ephemeral/prep/${SYSTEM_NAME}/networks/CAN.yaml - ip_address: 10.102.11.2
name: can-switch-1
comment: ""
aliases: []
- ip_address: 10.102.11.3
name: can-switch-2
comment: ""
aliases: []
net-name: CAN
vlan_id: 7
comment: ""
gateway: 10.102.11.1
The following is an example of spine switch IP addressing based on the network YAML example excerpts from above.
| VLAN | sw-spine-001 |
sw-spine-002 |
Purpose |
|---|---|---|---|
| 2 | 10.252.0.2/17 |
10.252.0.3/17 |
River node management |
| 4 | 10.254.0.2/17 |
10.254.0.3/17 |
River hardware management |
| 7 | 10.102.11.2/24 |
10.102.11.3/24 |
Customer access |
This should be set for every VLAN interface (1, 2, 4, 7, 10).
For more information, see How To Configure MAGP on Mellanox Switches.
Enable the MAGP protocol on sw-spine-01.
sw-spine-001 (config) # protocol magp
Enable the MAGP protocol on sw-spine-02.
Repeat the same commands from the previous step on sw-spine-02.
IP-Helpers will reside on VLANs 1, 2, 4, and 7.
Add DHCP relay configuration on sw-spine-01.
sw-spine-001 (config) #
ip dhcp relay instance 2 vrf default
ip dhcp relay instance 4 vrf default
ip dhcp relay instance 2 address 10.92.100.222
ip dhcp relay instance 4 address 10.94.100.222
interface vlan 1 ip dhcp relay instance 2 downstream
interface vlan 2 ip dhcp relay instance 2 downstream
interface vlan 4 ip dhcp relay instance 4 downstream
interface vlan 7 ip dhcp relay instance 2 downstream
Add DHCP relay configuration on sw-spine-02.
Repeat the same commands from the previous step on sw-spine-02.
OSPF is a dynamic routing protocol used to exchange routes.
It provides reachability from the MTN networks to NMN/Kubernetes networks.
The router-id used here is the NMN IP address (VLAN 2 IP address).
Configure OSPF.
Configure OSPF on sw-spine-01.
sw-spine-01 (config) #
protocol ospf
router ospf 1 vrf default
interface vlan 2 ip ospf area 0.0.0.2
interface vlan 4 ip ospf area 0.0.0.4
interface vlan 2 ip ospf priority 254
interface vlan 4 ip ospf priority 254
Configure OSPF on sw-spine-02.
Use the same commands as in the previous step.
Set the NMN VLAN configuration.
Set the NMN VLAN configuration on sw-spine-01.
sw-spine-001(config) #
vlan 2
interface vlan 2
interface vlan 2 ip address 10.252.0.2/17 primary
interface vlan 2 ipv4 port access-group nmn-hmn
interface vlan 2 ip ospf area 0.0.0.2
interface vlan 2 ip ospf priority 254
interface vlan 2 ip dhcp relay instance 2 downstream
interface vlan 2 magp 2
interface vlan 2 magp 2 ip virtual-router address 10.252.0.1
interface vlan 2 magp 2 ip virtual-router mac-address 00:00:5E:00:01:02
Set the NMN VLAN configuration on sw-spine-02.
sw-spine-002(config) #
vlan 2
interface vlan 2
interface vlan 2 ip address 10.252.0.3/17 primary
interface vlan 2 ipv4 port access-group nmn-hmn
interface vlan 2 ip ospf area 0.0.0.2
interface vlan 2 ip ospf priority 254
interface vlan 2 ip dhcp relay instance 2 downstream
interface vlan 2 magp 2
interface vlan 2 magp 2 ip virtual-router address 10.252.0.1
interface vlan 2 magp 2 ip virtual-router mac-address 00:00:5E:00:01:02
Set the HMN VLAN configuration.
Set the HMN VLAN configuration on sw-spine-001.
sw-spine-001(config) #
vlan 4
interface vlan 4
interface vlan 4 ip address 10.254.0.2/17 primary
interface vlan 4 ipv4 port access-group nmn-hmn
interface vlan 4 ip ospf area 0.0.0.4
interface vlan 4 ip ospf priority 254
interface vlan 4 ip dhcp relay instance 4 downstream
interface vlan 4 magp 4
interface vlan 4 magp 4 ip virtual-router address 10.254.0.1
interface vlan 4 magp 4 ip virtual-router mac-address 00:00:5E:00:01:04
Set the HMN VLAN configuration on sw-spine-002.
sw-spine-002(config) #
vlan 4
interface vlan 4
interface vlan 4 ip address 10.254.0.3/17 primary
interface vlan 4 ipv4 port access-group nmn-hmn
interface vlan 4 ip ospf area 0.0.0.4
interface vlan 4 ip ospf priority 254
interface vlan 4 ip dhcp relay instance 4 downstream
interface vlan 4 magp 4
interface vlan 4 magp 4 ip virtual-router address 10.254.0.1
interface vlan 4 magp 4 ip virtual-router mac-address 00:00:5E:00:01:04
Set the CAN VLAN configuration.
Set the CAN VLAN configuration on sw-spine-001.
sw-spine-001(config) #
vlan 7
interface vlan 7 ip address 10.101.8.2/24 primary
interface vlan 7 ip dhcp relay instance 2 downstream
interface vlan 7 magp 7
interface vlan 7 magp 7 ip virtual-router address 10.101.8.1
interface vlan 7 magp 7 ip virtual-router mac-address 00:00:5E:00:01:07
Set the CAN VLAN configuration on sw-spine-002.
sw-spine-002(config) #
vlan 7
interface vlan 7 ip address 10.101.8.3/24 primary
interface vlan 7 ip dhcp relay instance 2 downstream
interface vlan 7 magp 7
interface vlan 7 magp 7 ip virtual-router address 10.101.8.1
interface vlan 7 magp 7 ip virtual-router mac-address 00:00:5E:00:01:07
These two ports are cabled between the Mellanox switches.
Configure MLAG on sw-spine-001.
sw-spine-001(config) # protocol mlag
(config) # interface port-channel 100
(config) # interface ethernet 1/14 channel-group 100 mode active
(config) # interface ethernet 1/13 channel-group 100 mode active
(config) # interface ethernet 1/13 dcb priority-flow-control mode on force
(config) # interface ethernet 1/14 dcb priority-flow-control mode on force
(config) # vlan 4000
(config) # interface vlan 4000
(config) # interface port-channel 100 ipl 1
(config) # interface port-channel 100 dcb priority-flow-control mode on force
(config interface vlan 4000) # ip address 192.168.255.254 255.255.255.252
(config interface vlan 4000) # ipl 1 peer-address 192.168.255.253
(config) # mlag system-mac 00:00:5E:00:01:5D
(config) # no mlag shutdown
Configure MLAG on sw-spine-002.
sw-spine-002(config) # protocol mlag
(config) # interface port-channel 100
(config) # interface ethernet 1/14 channel-group 100 mode active
(config) # interface ethernet 1/13 channel-group 100 mode active
(config) # interface ethernet 1/13 dcb priority-flow-control mode on force
(config) # interface ethernet 1/14 dcb priority-flow-control mode on force
(config) # vlan 4000
(config) # interface vlan 4000
(config) # interface port-channel 100 ipl 1
(config) # interface port-channel 100 dcb priority-flow-control mode on force
(config interface vlan 4000) # ip address 192.168.255.253 255.255.255.252
(config interface vlan 4000) # ipl 1 peer-address 192.168.255.254
(config) # mlag system-mac 00:00:5E:00:01:5D
(config) # no mlag shutdown
These ports go to NCNs/UANs/switch downlinks.
Add MLAG ports to sw-spine-001.
sw-spine-001(config) # int mlag-port-channel 1
(config interface mlag-port-channel 1) # mtu 9216 force
(config interface mlag-port-channel 1) # switchport mode hybrid
(config interface mlag-port-channel 1) # no shutdown
(config interface mlag-port-channel 1) # lacp-individual enable force
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 2
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 4
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 7
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 10
Add MLAG ports to sw-spine-002.
NOTE: LACP fallback is only on one of the spines; disable lacp-individual enable force on sw-spine-002 if it was set previously.
sw-spine-002(config)# int mlag-port-channel 1
(config interface mlag-port-channel 1) # mtu 9216 force
(config interface mlag-port-channel 1) # switchport mode hybrid
(config interface mlag-port-channel 1) # no shutdown
(config interface mlag-port-channel 1) # no lacp-individual enable force
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 2
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 4
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 7
(config interface mlag-port-channel 1) # switchport hybrid allowed-vlan add 10
(config) # interface ethernet 1/1
(config interface ethernet 1/1) # mlag-channel-group 1 mode active
(config interface ethernet 1/1) # interface ethernet 1/1 speed 40G force
(config interface ethernet 1/1) # interface ethernet 1/1 mtu 9216 force
mgmt0 ports on both switches.NOTE: In the following examples, replace the SYSTEM_NAME value with the actual name of the system.
Configure MLAG-VIP on sw-spine-001.
sw-spine-001(config) #
no interface mgmt0 dhcp
interface mgmt0 ip address 192.168.255.241 /29
no mlag shutdown
mlag system-mac 00:00:5E:00:01:5D
mlag-vip SYSTEM_NAME-mlag-domain ip 192.168.255.242 /29 force
Configure MLAG-VIP on sw-spine-002.
sw-spine-002(config) #
no interface mgmt0 dhcp
interface mgmt0 ip address 192.168.255.243 /29
no mlag shutdown
mlag system-mac 00:00:5E:00:01:5D
mlag-vip SYSTEM_NAME-mlag-domain ip 192.168.255.242 /29 force
Verify the mlag-vip.
sw-spine-001 [SYSTEM_NAME-mlag-domain: master] # show mlag-vip
Example output:
MLAG-VIP:
MLAG group name: SYSTEM_NAME-mlag-domain
MLAG VIP address: 192.168.255.242/29
Active nodes: 2
----------------------------------------------------------------------------------
Hostname VIP-State IP Address
----------------------------------------------------------------------------------
sw-spine-001 master 192.168.255.241
sw-spine-002 standby 192.168.255.243
These ACLs are designed to block traffic from the Node Management Network (NMN) to and from the Hardware Management Network (HMN).
Create the access list.
NOTE: The following are examples only. The IP addresses below need to match what was generated by CSI.
sw-spine-001 & sw-spine-002 (config)#
sw-spine-001(config) # ipv4 access-list nmn-hmn
sw-spine-001(config ipv4 access-list nmn-hmn) # bind-point rif
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 10 deny ip 10.252.0.0 mask 255.255.128.0 10.254.0.0 mask 255.255.128.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 20 deny ip 10.252.0.0 mask 255.255.128.0 10.104.0.0 mask 255.252.0.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 30 deny ip 10.254.0.0 mask 255.255.128.0 10.252.0.0 mask 255.255.128.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 40 deny ip 10.254.0.0 mask 255.255.128.0 10.100.0.0 mask 255.252.0.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 50 deny ip 10.100.0.0 mask 255.252.0.0 10.254.0.0 mask 255.255.128.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 60 deny ip 10.100.0.0 mask 255.252.0.0 10.104.0.0 mask 255.252.0.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 70 deny ip 10.104.0.0 mask 255.252.0.0 10.252.0.0 mask 255.255.128.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 80 deny ip 10.104.0.0 mask 255.252.0.0 10.100.0.0 mask 255.252.0.0
sw-spine-001(config ipv4 access-list nmn-hmn) # seq-number 90 permit ip any any
sw-spine-001(config ipv4 access-list nmn-hmn) # exit
Apply ACL to VLANs.
sw-spine-001(config) # interface vlan 2 ipv4 port access-group nmn-hmn
sw-spine-001(config) # interface vlan 4 ipv4 port access-group nmn-hmn
Spanning-tree will need to be applied to each MAGP pair. sw-spine-001 will have a lower priority making it the root bridge.
Apply the following configuration to the Mellanox spine switches.
This is an example of a switch-to-switch connection.
sw-spine-001 & sw-spine-002 (config)#
spanning-tree mode rpvst
spanning-tree port type edge default
interface ethernet 1/13-1/14 spanning-tree port type network
interface ethernet 1/15/1-1/15/2 spanning-tree port type network
interface mlag-port-channel 113 spanning-tree port type network
interface mlag-port-channel 151-152 spanning-tree port type network
interface ethernet 1/13-1/14 spanning-tree guard root
interface ethernet 1/15/1-1/15/2 spanning-tree guard root
interface mlag-port-channel 113 spanning-tree guard root
interface mlag-port-channel 151-152 spanning-tree guard root
spanning-tree port type edge bpdufilter default
spanning-tree port type edge bpduguard default
spanning-tree vlan 1-2 priority 0
spanning-tree vlan 4 priority 0
spanning-tree vlan 7 priority 0
spanning-tree vlan 10 priority 0
The IP addresses used here will be the first three worker nodes on the NMN network. These can be found in NMN.yaml.
Get current NTP configuration.
sw-spine-001 [standalone: master] (config) # show running-config | include ntp
no ntp server 10.252.1.9 disable
ntp server 10.252.1.9 keyID 0
no ntp server 10.252.1.9 trusted-enable
ntp server 10.252.1.9 version 4
no ntp server 10.252.1.10 disable
ntp server 10.252.1.10 keyID 0
no ntp server 10.252.1.10 trusted-enable
ntp server 10.252.1.10 version 4
no ntp server 10.252.1.11 disable
ntp server 10.252.1.11 keyID 0
no ntp server 10.252.1.11 trusted-enable
ntp server 10.252.1.11 version 4
Delete any current NTP configuration.
sw-spine-001# configure terminal
sw-spine-001 [standalone: master] (config) # no ntp server 10.252.1.9
sw-spine-001 [standalone: master] (config) # no ntp server 10.252.1.10
sw-spine-001 [standalone: master] (config) # no ntp server 10.252.1.11
Add new NTP server configuration.
sw-spine-001 [standalone: master] (config) # ntp server 10.252.1.12
sw-spine-001 [standalone: master] (config) # ntp server 10.252.1.13
sw-spine-001 [standalone: master] (config) # ntp server 10.252.1.14
Verify NTP status.
sw-spine-001 [standalone: master] # show ntp
Example output:
NTP is administratively : enabled
NTP Authentication administratively: disabled
NTP server role : enabled
Clock is synchronized:
Reference: 10.252.1.14
Offset : -0.056 ms
Active servers and peers:
10.252.1.12:
Conf Type : serv
Status : candidat(+)
Stratum : 4
Offset(msec) : -0.119
Ref clock : 10.252.1.4
Poll Interval (sec): 128
Last Response (sec): 107
Auth state : none
10.252.1.13:
Conf Type : serv
Status : candidat(+)
Stratum : 4
Offset(msec) : -0.059
Ref clock : 10.252.1.4
Poll Interval (sec): 128
Last Response (sec): 96
Auth state : none
10.252.1.14:
Conf Type : serv
Status : sys.peer(*)
Stratum : 4
Offset(msec) : -0.056
Ref clock : 10.252.1.4
Poll Interval (sec): 128
Last Response (sec): 118
Auth state : none
Configure DNS.
This will point to the unbound DNS server.
sw-spine-001 & sw-spine-002 (config)#
ip name-server 10.92.100.225
Verify the configuration.
sw-spine-002 [standalone: master] # show ip dhcp relay
Example output:
Instance ID 2:
VRF Name: default
DHCP Servers:
10.92.100.222
DHCP relay agent options:
always-on : Disabled
Information Option: Disabled
UDP port : 67
Auto-helper : Disabled
-------------------------------------------
Interface Label Mode
-------------------------------------------
vlan1 N/A downstream
vlan2 N/A downstream
vlan7 N/A downstream
Instance ID 4:
VRF Name: default
DHCP Servers:
10.94.100.222
DHCP relay agent options:
always-on : Disabled
Information Option: Disabled
UDP port : 67
Auto-helper : Disabled
-------------------------------------------
Interface Label Mode
-------------------------------------------
vlan4 N/A downstream
Save the changes made during this configuration session.
sw-spine-001(config)# exit
sw-spine-001# write memory
Show the current configuration
sw-spine-001# show running-config