CSM 1.7.0 adds support for IPv6 on the Customer Management Network (CMN), and Customer High-Speed Network (CHN).
This functionality is limited in scope:
bond0.cmn0 interface on the NCNs.ncn-m001 lan0 network interface is supported.cray-dns-unbound service can be configured to access a site DNS server using IPv6 over the CMN.cray-keycloak service can be configured to access an LDAP server using IPv6 over the CMN.The CMN and CAN networks in the System Layout Service (SLS)
have CIDR6, Gateway6, and IPAddress6 fields added to avoid overlap with existing IPv4 data.
Example:
{
"Name": "CMN",
"FullName": "Customer Management Network",
"IPRanges": [
"10.102.193.0/25"
],
"Type": "ethernet",
"ExtraProperties": {
"CIDR": "10.102.193.0/25",
"CIDR6": "2001:db8:100:200::/64",
"MTU": 9000,
"MyASN": 65532,
"PeerASN": 65533,
"Subnets": [
{
"CIDR": "10.102.193.0/25",
"CIDR6": "2001:db8:100:200::/64",
"FullName": "CMN Management Network Infrastructure",
"Gateway": "10.102.193.1",
"Gateway6": "2001:db8:100:200::1",
"IPReservations": [
{
"Comment": "x3000c0h12s1",
"IPAddress": "10.102.193.2",
"IPAddress6": "2001:db8:100:200::2",
"Name": "sw-spine-001"
}
]
}
]
}
}
In the Boot Script Service (BSS),
the cloud-init metadata for each NCN has ip6 and gateway6 fields added.
This allows IPv6 to be configured when NCNs are rebuilt.
Example:
{
"cloud-init": {
"meta-data": {
"availability-zone": "x3000",
"instance-id": "i-899A1802",
"ipam": {
"cmn": {
"gateway": "10.102.193.1",
"gateway6": "fdf8:413:de2c:200::1",
"ip": "10.102.193.40/25",
"ip6": "fdf8:413:de2c:200::108/64",
"parent_device": "bond0",
"vlanid": 7
},
"hmn": {
"gateway": "10.254.0.1",
"ip": "10.254.1.17/17",
"parent_device": "bond0",
"vlanid": 4
},
"mtl": {
"gateway": "10.1.0.1",
"ip": "10.1.1.8/16",
"parent_device": "bond0",
"vlanid": 0
},
"nmn": {
"gateway": "10.252.0.1",
"ip": "10.252.1.10/17",
"parent_device": "bond0",
"vlanid": 2
}
}
}
}
}
IPv6 support can be enabled in two different ways.
IPv6 can be enabled as part of a fresh install of CSM.
During the install, if an administrator wishes to enable IPv6, this must be done during the Create system configuration procedure.
New command line options were added to the Cray Site Initializer tool (csi).
| Option | Description |
|---|---|
chn-gateway6 |
IPv6 Gateway for NCNs on the CHN |
chn-cidr6 |
Overall IPv6 CIDR for all Customer High-Speed subnets |
cmn-gateway6 |
Overall IPv6 CIDR for all Customer Management subnets |
cmn-cidr6 |
IPv6 Gateway for NCNs on the CMN |
These options can be used during a fresh install to configure IPv6. See
cray-site-init updates
for more information.
During the install, administrators may optionally configure some services for IPv6
during the Prepare site-init
procedure.
IPv6 can be enabled as part of an upgrade from CSM 1.6 to CSM 1.7. It cannot be enabled as part of a CSM 1.7 to CSM 1.7 patch upgrade.
During the upgrade, if an administrator wishes to enable IPv6, this must be done during at the beginning of the
Execute the IUF management-nodes-rollout stage.
This ensures that the NCNs are rebuilt with IPv6 support enabled.
A new patch subcommand as been added to csi. The csi patch csm ipv6 command takes the
chn-gateway6, chn-cidr6, cmn-gateway6, and cmn-cidr6 arguments described in
Enabling IPv6 during CSM install. It uses that
information to update SLS and BSS with the IPv6 data.
This command defaults to a dry run and writes all proposed BSS and SLS changes, along with
backups of the original data. By default this backup is to a timestamped directory in the
current working directory. This behavior can be overridden with the -b|--backup-dir option.
The --commit option will apply the proposed changes to BSS and SLS.
See cray-site-init updates for a detailed description of the csi patch csm ipv6 options.
During the upgrade, administrators may optionally configure some services for IPv6.
This is also done at the beginning of the
Execute the IUF management-nodes-rollout stage,
after IPv6 has been enabled.
The CSM Automatic Network Utility (CANU) will automatically generate configuration with IPv6 support enabled when supplied an SLS file with IPv6 entries.
See the CSM Automatic Network Utility documentation for more information on network configuration generation and validation.
CANU only generates the networking configuration required by CSM, it does not configure any routes out of the spine switches to site networks. External connectivity can configured by means of a CANU custom configuration file. There are many ways in which external connectivity can be achieved and discussing these options is beyond the scope of this document. See Connect to the CMN and CAN for some suggestions. Administrators are encouraged to consult their site networking team, in order to design the best solution for the site.
Several CSM services can be configured to use IPv6.
The cray-dns-unbound service can be configured to access a site DNS server using IPv6. See cray-dns-unbound IPv6 Support for more information.
IMPORTANT IPv6 must have been configured and enabled on the CMN and NCNs before this is enabled otherwise DNS queries may timeout resulting in system instability.
The cray-keycloak service can be configured to access an LDAP server using IPv6. See Keycloak IPv6 Support for more information.
If using LDAP over SSL, then the IPv6 address or hostname used must be present as a Subject Alternative Name in the LDAP server certificate, otherwise access will fail because of certificate verification issues.
An IPv6 address can be used to define an NTP time source. See Configure NTP on NCNs for information on how to reconfigure a running system.
If performing a fresh install simply add the IP address or hostname to ntp-servers in system_config.yaml.
uan_can_setup option must be enabled in the uss-config-management VCS repo in order to apply this configuration to the node.
See the HPE Cray Supercomputing User Services Software (USS) for more information.