Cray System Management Documentation > Install CSM > Prepare site-init

Prepare `site-init`

These procedures guide administrators through setting up the site-init directory which contains important customizations for various products.

Background
Create and initialize site-init directory
Create baseline system customizations
- Setup LDAP configuration
- After LDAP configuration
Customer-specific customizations
Version control site-init files
1. Push to a remote repository

1. Background

The shasta-cfg directory included in the CSM release tarball includes relatively static, installation-centric artifacts, such as:

Cluster-wide network configuration settings required by Helm charts deployed by product stream Loftsman manifests
Sealed Secrets
Sealed Secret Generate Blocks – a form of plain-text input that renders to a Sealed Secret
Helm chart value overrides that are merged into Loftsman manifests by product stream installers

2. Create and initialize `site-init` directory

Set the SITE_INIT variable.

Important: All procedures on this page assume that SITE_INIT variable has been set.
```
pit# SITE_INIT="${PITDATA}/prep/site-init"
```
Create the site-init directory.
```
pit# mkdir -pv "${SITE_INIT}"
```

Initialize site-init from CSM.

pit# "${CSM_PATH}/shasta-cfg/meta/init.sh" "${SITE_INIT}"

3. Create baseline system customizations

The following steps update ${SITE_INIT}/customizations.yaml with system-specific customizations.

Change into the site-init directory
```
pit# cd "${SITE_INIT}"
```

Merge the system-specific settings generated by CSI into customizations.yaml.

pit# yq merge -xP -i "${SITE_INIT}/customizations.yaml" <(yq prefix -P "${PITDATA}/prep/${SYSTEM_NAME}/customizations.yaml" spec)

Set the cluster name.

pit# yq write -i "${SITE_INIT}/customizations.yaml" spec.wlm.cluster_name "${SYSTEM_NAME}"

Make a backup copy of ${SITE_INIT}/customizations.yaml.

pit# cp -pv "${SITE_INIT}/customizations.yaml" "${SITE_INIT}/customizations.yaml.prepassword"

Review the configuration to generate these sealed secrets in customizations.yaml in the site-init directory:
- spec.kubernetes.sealed_secrets.cray_reds_credentials
- spec.kubernetes.sealed_secrets.cray_meds_credentials
- spec.kubernetes.sealed_secrets.cray_hms_rts_credentials
- Replace the Username and Password references in match the existing settings of your system hardware components.
NOTE
- The cray_reds_credentials are used by the River Endpoint Discovery Service (REDS) for River components.
- The cray_meds_credentials are used by the Mountain Endpoint Discovery Service (MEDS) for the liquid-cooled components in an Olympus (Mountain) cabinet.
- The cray_hms_rts_credentials are used by the Redfish Translation Service (RTS) for any hardware components which are not managed by Redfish, such as a ServerTech PDU in a River Cabinet.
See the Decrypt Sealed Secrets for Review section of Manage Sealed Secrets, if needing to examine credentials from prior installations.
```
pit# vim "${SITE_INIT}/customizations.yaml"
```

Review the changes that you made.

pit# diff ${SITE_INIT}/customizations.yaml ${SITE_INIT}/customizations.yaml.prepassword

Validate that REDS/MEDS/RTS credentials are correct.

For all credentials, make sure that Username and Password values are correct.
- Validate REDS credentials:
  NOTE These credentials are used by the REDS and HMS discovery services, targeting River Redfish BMC endpoints and management switches
  - For vault_redfish_defaults, the only entry used is:
    {"Cray": {"Username": "root", "Password": "XXXX"}
  - Ensure the Cray key exists. This key is not used in any of the other credential specifications.
```
pit# yq read "${SITE_INIT}/customizations.yaml" 'spec.kubernetes.sealed_secrets.cray_reds_credentials.generate.data[*].args.value' | jq
```
- Validate MEDS credentials:
  
  These credentials are used by the MEDS service, targeting Redfish BMC endpoints.
```
pit# yq read "${SITE_INIT}/customizations.yaml" 'spec.kubernetes.sealed_secrets.cray_meds_credentials.generate.data[0].args.value' | jq
```
- Validate RTS credentials:
  
  These credentials are used by the Redfish Translation Service, targeting River Redfish BMC endpoints and PDU controllers.
```
pit# yq read "${SITE_INIT}/customizations.yaml" 'spec.kubernetes.sealed_secrets.cray_hms_rts_credentials.generate.data[*].args.value' | jq
```
To customize the PKI Certificate Authority (CA) used by the platform, see Certificate Authority.

IMPORTANT The CA may not be modified after install.

Setup LDAP configuration

NOTE Skip ahead to After LDAP configuration if there is no LDAP configuration at this time. If LDAP should be enabled later, follow Add LDAP User Federation after installation.

Set environment variables for the LDAP server and its port.

In the example below, the LDAP server has the hostname dcldap2.hpc.amslabs.hpecorp.net and is using the port 636.
```
pit# LDAP=dcldap2.hpc.amslabs.hpecorp.net
pit# PORT=636
```

Load the openjdk container image.

NOTE Requires a properly configured Docker or Podman environment.

pit# "${CSM_PATH}/hack/load-container-image.sh" artifactory.algol60.net/csm-docker/stable/docker.io/library/openjdk:11-jre-slim

Get the issuer certificate.

Retrieve the issuer certificate for the LDAP server at port 636. Use openssl s_client to connect and show the certificate chain returned by the LDAP host:
```
pit# openssl s_client -showcerts -connect "${LDAP}:${PORT}" </dev/null
```
Enter the issuer’s certificate into cacert.pem.

Either manually extract (i.e., cut/paste) the issuer’s certificate into cacert.pem, or try the following commands to create it automatically.

NOTE The following commands were verified using OpenSSL version 1.1.1d and use the -nameopt RFC2253 option to ensure consistent formatting of distinguished names. Unfortunately, older versions of OpenSSL may not support -nameopt on the s_client command or may use a different default format. However, the issuer certificate can be manually extracted from the output of the above openssl s_client example, if the following commands are unsuccessful.
1. Observe the issuer’s DN.
```
pit# openssl s_client -showcerts -nameopt RFC2253 -connect "${LDAP}:${PORT}" </dev/null 2>/dev/null |
        grep issuer= | sed -e 's/^issuer=//'
```
  Expected output includes a line similar to this:
```
emailAddress=dcops@hpe.com,CN=Data Center,OU=HPC/MCS,O=HPE,ST=WI,C=US
```
2. Extract the issuer’s certificate.
  
  NOTE The issuer DN is properly escaped as part of the awk pattern below. It must be changed to match the value for emailAddress, CN, OU, etc. for your LDAP. If the value you are using is different, be sure to escape it properly!
```
pit# openssl s_client -showcerts -nameopt RFC2253 -connect "${LDAP}:${PORT}" </dev/null 2>/dev/null |
          awk '/s:emailAddress=dcops@hpe.com,CN=Data Center,OU=HPC\/MCS,O=HPE,ST=WI,C=US/,/END CERTIFICATE/' |
          awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/' > cacert.pem
```

Create certs.jks.

NOTE The alias used in this command for cray-data-center-ca should be changed to match your LDAP.

pit# podman run --rm -v "$(pwd):/data" \
        artifactory.algol60.net/csm-docker/stable/docker.io/library/openjdk:11-jre-slim keytool \
        -importcert -trustcacerts -file /data/cacert.pem -alias cray-data-center-ca \
        -keystore /data/certs.jks -storepass password -noprompt

Create certs.jks.b64 by base-64 encoding certs.jks.
```
pit# base64 certs.jks > certs.jks.b64
```

Inject and encrypt certs.jks.b64 into customizations.yaml.

pit# cat <<EOF | yq w - 'data."certs.jks"' "$(<certs.jks.b64)" | \
        yq r -j - | ${SITE_INIT}/utils/secrets-encrypt.sh | \
        yq w -f - -i ${SITE_INIT}/customizations.yaml 'spec.kubernetes.sealed_secrets.cray-keycloak'
{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "keycloak-certs",
    "namespace": "services",
    "creationTimestamp": null
  },
  "data": {}
}
EOF

Update the keycloak_users_localize sealed secret with the appropriate value for ldap_connection_url.

Set ldap_connection_url in customizations.yaml.

pit# yq write -i "${SITE_INIT}/customizations.yaml" \
         'spec.kubernetes.sealed_secrets.keycloak_users_localize.generate.data.(args.name==ldap_connection_url).args.value' \
         "ldaps://${LDAP}"

Review the keycloak_users_localize sealed secret.

pit# yq read "${SITE_INIT}/customizations.yaml" spec.kubernetes.sealed_secrets.keycloak_users_localize

Configure the ldapSearchBase and localRoleAssignments settings for the cray-keycloak-users-localize chart in customizations.yaml.

NOTE There may be one or more groups in LDAP for admins and one or more for users. Each admin group needs to be assigned to role admin and set to both shasta and cray clients in Keycloak. Each user group needs to be assigned to role user and set to both shasta and cray clients in Keycloak.

Set ldapSearchBase in customizations.yaml.

NOTE This example sets ldapSearchBase to dc=dcldap,dc=dit

pit# yq write -i "${SITE_INIT}/customizations.yaml" spec.kubernetes.services.cray-keycloak-users-localize.ldapSearchBase 'dc=dcldap,dc=dit'

Set localRoleAssignments in customizations.yaml.

NOTE This example sets localRoleAssignments for the LDAP groups employee, craydev, and shasta_admins to be the admin role, and the LDAP group shasta_users to be the user role.

pit# yq write -s - -i "${SITE_INIT}/customizations.yaml" <<EOF
- command: update
  path: spec.kubernetes.services.cray-keycloak-users-localize.localRoleAssignments
  value:
  - {"group": "employee", "role": "admin", "client": "shasta"}
  - {"group": "employee", "role": "admin", "client": "cray"}
  - {"group": "craydev", "role": "admin", "client": "shasta"}
  - {"group": "craydev", "role": "admin", "client": "cray"}
  - {"group": "shasta_admins", "role": "admin", "client": "shasta"}
  - {"group": "shasta_admins", "role": "admin", "client": "cray"}
  - {"group": "shasta_users", "role": "user", "client": "shasta"}
  - {"group": "shasta_users", "role": "user", "client": "cray"}
EOF

Review the cray-keycloak-users-localize values.

pit# yq read "${SITE_INIT}/customizations.yaml" spec.kubernetes.services.cray-keycloak-users-localize

After LDAP configuration

Configure the Unbound DNS resolver (if needed).

Important If access to a site DNS server is required and this DNS server was specified to csi using the site-dns option (either on the command line or in the system_config.yaml file), then no further action is required and this step should be skipped.

The default configuration is as follows:
```
cray-dns-unbound:
    domain_name: '{{ network.dns.external }}'
    forwardZones:
      - name: "."
        forwardIps:
          - "{{ network.netstaticips.system_to_site_lookups }}"
```
The configured site DNS server can be verified by inspecting the value set for system_to_site_lookups.
```
pit# yq r ${SITE_INIT}/customizations.yaml spec.network.netstaticips.system_to_site_lookups
```
Possible output:
```
172.30.84.40
```
If there is no requirement to resolve external hostnames (including other services on the site network) or no upstream DNS server, then the cray-dns-unbound service should be configured to forward to the cray-dns-powerdns service.
1. Update the forwardZones configuration for the cray-dns-unbound service to point to the cray-dns-powerdns service and only answer mtl network queries from static records.
```
pit# yq write -s - -i ${SITE_INIT}/customizations.yaml <<EOF
- command: update
  path: spec.kubernetes.services.cray-dns-unbound.forwardZones
  value:
  - name: "."
    forwardIps:
    - "10.92.100.85"
- command: update
  path: spec.kubernetes.services.cray-dns-unbound.localZones
  value:
  - name: "mtl."
    localType: static
EOF
```
2. Review the cray-dns-unbound values.
  
  IMPORTANT Do not remove the domain_name entry, it is required for Unbound to forward requests to PowerDNS correctly.
```
pit# yq read "${SITE_INIT}/customizations.yaml" spec.kubernetes.services.cray-dns-unbound
```
  Expected output:
```
domain_name: '{{ network.dns.external }}'
forwardZones:
  - name: "."
    forwardIps:
      - "10.92.100.85"
localZones:
  - name: "mtl."
    localType: static
```
See the following documentation regarding known issues when operating with no upstream DNS server.
- Spire Database Cluster DNS Lookup Failure
- Spire database connection pool configuration in an air-gapped environment
(Optional) Configure PowerDNS zone transfer and DNSSEC. See the PowerDNS Configuration Guide for more information.
- If zone transfer is to be configured, then review customizations.yaml and ensure that the primary_server, secondary_servers, and notify_zones values are set correctly.
- If DNSSEC is to be used, then add the desired keys into the dnssec SealedSecret.
(Optional) Configure Prometheus SNMP Exporter.

The Prometheus SNMP exporter needs to be configured with a list of management network switches to scrape metrics from in order to populate the System Health Service Grafana dashboards.

See Prometheus SNMP Exporter for more information.
Load the zeromq container image required by Sealed Secret Generators.

NOTE Requires a properly configured Docker or Podman environment.
```
pit# "${CSM_PATH}/hack/load-container-image.sh" artifactory.algol60.net/csm-docker/stable/docker.io/zeromq/zeromq:v4.0.5
```

Re-encrypt existing secrets.

pit# "${SITE_INIT}/utils/secrets-reencrypt.sh" \
        "${SITE_INIT}/customizations.yaml" \
        "${SITE_INIT}/certs/sealed_secrets.key" \
        "${SITE_INIT}/certs/sealed_secrets.crt"

It is not an error if this script gives no output.

Generate secrets.

pit# "${SITE_INIT}/utils/secrets-seed-customizations.sh" "${SITE_INIT}/customizations.yaml"

Leave the site-init directory.
```
pit# cd "${PITDATA}"
```
site-init is now prepared. Resume Initialize the LiveCD.

4. Customer-specific customizations

Customer-specific customizations are any changes on top of the baseline configuration to satisfy customer-specific requirements. It is recommended that customer-specific customizations be tracked on branches separate from the mainline in order to make them easier to manage.

Apply any customer-specific customizations by merging the corresponding branches into master branch of site-init.

When considering merges, and especially when resolving conflicts, carefully examine differences to ensure all changes are relevant. For example, when applying a customer-specific customization used in a prior version, be sure the change still makes sense. It is common for options to change as new features are introduced and bugs are fixed.

5. Version control `site-init` files

Setup site-init as a Git repository in order to manage the baseline configuration during initial system installation.

Initialize site-init as a Git repository.
```
pit# cd ${SITE_INIT}
pit# git init .
```
(Optional) Exclude sealed secret private keys from Git.

WARNING If production system or operational security is a concern, do NOT store the sealed secret private key in Git; instead, store the sealed secret key outside of Git in a secure offline system. In order to ensure that these sensitive keys are not accidentally committed, configure .gitignore to ignore files under the certs directory:
```
pit# echo "certs/" >> .gitignore
```
Stage site-init files to be committed.
```
pit# git add -A
```
Review what will be committed.
```
pit# git status
```

Commit the baseline configuration.

pit# git commit -m "Baseline configuration for $(${CSM_PATH}/lib/version.sh)"

5.1 Push to a remote repository

It is strongly recommended that the site-init repository be maintained off-cluster. Add a remote repository and push the baseline configuration on master branch to a corresponding remote branch.

Prepare site-init