Customize the Broker UAI Image

The Broker UAI image that comes with UAS is the image used to construct Broker UAIs.

The key pieces of the Broker UAI image are:

  • An entrypoint shell script that initializes the container and starts the SSH daemon running.
  • An SSH configuration that forces logged in users into the switchboard command which creates / selects End-User UAIs and redirects connections.

The primary way to customize the Broker UAI image is by defining volumes and connecting them to the Broker UAI class for a given broker. Some customizations may require action that cannot be covered simply by using volumes to override configuration. Those cases generally require changing the Broker UAI behavior in some way. Those cases can be covered either by volume mounting a customized entrypoint script, or volume mounting a customized SSH configuration. Both of these cases are shown in the following examples.

Customize the Broker UAI Entrypoint Script

The Broker UAI entrypoint script runs once every time the Broker UAI starts. It resides at /app/broker/entrypoint.sh in the Broker UAI image. The entrypoint script is the only file in that directory, so it can be overridden by creating a Kubernetes ConfigMap in the uas namespace containing the modified script and creating a volume using that ConfigMap with a mount point of /app/broker. There is critical content in the entrypoint script that should not be modified.

The following shows the contents of an unmodified script:

#!/bin/bash

# MIT License
#
# (C) Copyright [2020] Hewlett Packard Enterprise Development LP
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.

echo "Configure PAM to use sssd..."
pam-config -a --sss --mkhomedir

echo "Generating broker host keys..."
ssh-keygen -A

echo "Checking for UAI_CREATION_CLASS..."
if ! [ -z $UAI_CREATION_CLASS ]; then
    echo UAI_CREATION_CLASS=$UAI_CREATION_CLASS >> /etc/environment
fi

echo "Starting sshd..."
/usr/sbin/sshd -f /etc/switchboard/sshd_config

echo "Starting sssd..."
sssd

sleep infinity

Starting at the top:

  • pam_config ... can be customized to set up PAM as needed. The configuration here assumes the broker is using SSSD to reach a directory server for authentication and that, if a home directory is not present for a user at login, one should be made on the broker.
  • The ssh-keygen... part is needed to set up the SSH host key for the broker and should be left alone.
  • The UAI_CREATION_CLASS code should be left alone, as it sets up information used by switchboard to create End-User UAIs.
  • The /usr/sbin/sshd... part starts the SSH server on the broker and should be left alone. Configuration of SSH is covered in the next section and is done by replacing /etc/switchboard/sshd_config not by modifying this line.
  • The sssd part assumes the broker is using SSSD to reach a directory server, it can be changed as needed.
  • The sleep infinity prevents the script from exiting which keeps the Broker UAI running. It should not be removed or altered.

As long as the basic flow and contents described here are honored, other changes to this script should work without compromising the Broker UAI’s function.

The following is an example of replacing the entrypoint script with a new entrypoint script that changes the SSSD invocation to explicitly specify the sssd.conf file path (the standard path is used here, but a different path might make customizing SSSD for a given site simpler under some set of circumstances):

  1. Create a new entrypoint script.

    NOTE A special “here document” form is used to prevent variable substitution in the file.

    ncn-m001-cat <<-"EOF" > entrypoint.sh
    #!/bin/bash
    
    # MIT License
    #
    # (C) Copyright [2020] Hewlett Packard Enterprise Development LP
    #
    # Permission is hereby granted, free of charge, to any person obtaining a
    # copy of this software and associated documentation files (the "Software"),
    # to deal in the Software without restriction, including without limitation
    # the rights to use, copy, modify, merge, publish, distribute, sublicense,
    # and/or sell copies of the Software, and to permit persons to whom the
    # Software is furnished to do so, subject to the following conditions:
    #
    # The above copyright notice and this permission notice shall be included
    # in all copies or substantial portions of the Software.
    #
    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
    # THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
    # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
    # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
    # OTHER DEALINGS IN THE SOFTWARE.
    
    echo "Configure PAM to use sssd..."
    pam-config -a --sss --mkhomedir
    
    echo "Generating broker host keys..."
    ssh-keygen -A
    
    echo "Checking for UAI_CREATION_CLASS..."
    if ! [ -z $UAI_CREATION_CLASS ]; then
        echo UAI_CREATION_CLASS=$UAI_CREATION_CLASS >> /etc/environment
    fi
    
    echo "Starting sshd..."
    /usr/sbin/sshd -f /etc/switchboard/sshd_config
    
    echo "Starting sssd..."
    # LOCAL MODIFICATION
    # change the normal SSSD invocation
    # sssd
    # to specify the config file path
    sssd --config /etc/sssd/sssd.conf
    # END OF LOCAL MODIFICATION
    
    sleep infinity
    EOF
    
  2. Create a new ConfigMap with the content from the script.

    ncn-m001-kubectl create configmap -n uas broker-entrypoint --from-file=entrypoint.sh
    
  3. Create a new volume.

    NOTE: The default_mode setting, which will set the mode on the file /app/broker/entrypoint.sh is decimal 493 here instead of octal 0755. The octal notation is not permitted in a JSON specification. Decimal numbers have to be used.

    ncn-m001-cray uas admin config volumes create --mount-path /app/broker --volume-description '{"config_map": {"name": "broker-entrypoint", "default_mode": 493}}' --volumename broker-entrypoint
    

    Example output:

    mount_path = "/app/broker"
    volume_id = "2246bbb1-4006-4b11-ba57-6588a7b7c02f"
    volumename = "broker-entrypoint"
    
    [volume_description.config_map]
    default_mode = 493
    name = "broker-entrypoint"
    
  4. List the UAI classes.

    ncn-m001-cray uas admin config classes list | grep -e class_id -e comment
    

    Example output:

    class_id = "5eb523ba-a3b7-4a39-ba19-4cfe7d19d296"
    comment = "UAI Class to Create Non-Brokered End-User UAIs"
    class_id = "bdb4988b-c061-48fa-a005-34f8571b88b4"
    comment = "UAI Class to Create Brokered End-User UAIs"
    comment = "Resource Specification to use with Brokered End-User UAIs"
    class_id = "d764c880-41b8-41e8-bacc-f94f7c5b053d"
    comment = "UAI broker class"
    
  5. Describe the desired UAI class.

    ncn-m001-cray uas admin config classes describe d764c880-41b8-41e8-bacc-f94f7c5b053d --format yaml
    

    Example output:

    class_id: d764c880-41b8-41e8-bacc-f94f7c5b053d
    comment: UAI broker class
    default: false
    image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
    namespace: uas
    opt_ports: []
    priority_class_name: uai-priority
    public_ip: true
    replicas: 1
    resource_config:
    resource_id:
    service_account:
    timeout:
    tolerations:
    uai_compute_network: false
    uai_creation_class: bdb4988b-c061-48fa-a005-34f8571b88b4
    uai_image:
      default: false
      image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
      imagename: registry.local/cray/cray-uai-broker:1.2.4
    volume_list:
    - 11a4a22a-9644-4529-9434-d296eef2dc48
    - 1ec36af0-d5b6-4ad9-b3e8-755729765d76
    - a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
    volume_mounts:
    - mount_path: /etc/localtime
      volume_description:
        host_path:
          path: /etc/localtime
          type: FileOrCreate
      volume_id: 11a4a22a-9644-4529-9434-d296eef2dc48
      volumename: timezone
    - mount_path: /etc/sssd
      volume_description:
        secret:
          default_mode: 384
          secret_name: broker-sssd-conf
      volume_id: 1ec36af0-d5b6-4ad9-b3e8-755729765d76
      volumename: broker-sssd-config
    - mount_path: /lus
      volume_description:
        host_path:
          path: /lus
          type: DirectoryOrCreate
      volume_id: a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
      volumename: lustre
    
  6. Update the UAI class.

    ncn-m001-cray uas admin config classes update \
    --volume-list '11a4a22a-9644-4529-9434-d296eef2dc48,1ec36af0-d5b6-4ad9-b3e8-755729765d76,2246bbb1-4006-4b11-ba57-6588a7b7c02f,a3b149fd-c477-41f0-8f8d-bfcee87fdd0a' \
    d764c880-41b8-41e8-bacc-f94f7c5b053d --format yaml
    

    Example output:

    class_id: d764c880-41b8-41e8-bacc-f94f7c5b053d
    comment: UAI broker class
    default: false
    image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
    namespace: uas
    opt_ports: []
    priority_class_name: uai-priority
    public_ip: true
    replicas: 1
    resource_config:
    resource_id:
    service_account:
    timeout:
    tolerations:
    uai_compute_network: false
    uai_creation_class: bdb4988b-c061-48fa-a005-34f8571b88b4
    uai_image:
      default: false
      image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
      imagename: registry.local/cray/cray-uai-broker:1.2.4
    volume_list:
    - 11a4a22a-9644-4529-9434-d296eef2dc48
    - 1ec36af0-d5b6-4ad9-b3e8-755729765d76
    - 2246bbb1-4006-4b11-ba57-6588a7b7c02f
    - a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
    volume_mounts:
    - mount_path: /etc/localtime
      volume_description:
        host_path:
          path: /etc/localtime
          type: FileOrCreate
      volume_id: 11a4a22a-9644-4529-9434-d296eef2dc48
      volumename: timezone
    - mount_path: /etc/sssd
      volume_description:
        secret:
          default_mode: 384
          secret_name: broker-sssd-conf
      volume_id: 1ec36af0-d5b6-4ad9-b3e8-755729765d76
      volumename: broker-sssd-config
    - mount_path: /app/broker
      volume_description:
        config_map:
          default_mode: 493
          name: broker-entrypoint
      volume_id: 2246bbb1-4006-4b11-ba57-6588a7b7c02f
      volumename: broker-entrypoint
    - mount_path: /lus
      volume_description:
        host_path:
          path: /lus
          type: DirectoryOrCreate
      volume_id: a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
      volumename: lustre
    
  7. After the Broker UAI class is updated, all that remains is to clear out any existing End-User UAIs (existing UAIs will not work with the new broker because the new broker will have a new key-pair shared with its UAIs) and the existing Broker UAI (if any) and create a new Broker UAI.

    NOTE Clearing out existing UAIs will terminate any user activity on those UAIs, make sure that users are warned of the disruption.

    1. Clear out the UAIs.

      ncn-m001-cray uas admin uais delete --class-id bdb4988b-c061-48fa-a005-34f8571b88b4
      ncn-m001-cray uas admin uais delete --class-id d764c880-41b8-41e8-bacc-f94f7c5b053d
      

      Output similar to results = [ "Successfully deleted uai-vers-e937b810",] will be returned for each command.

    2. Restart the broker.

      ncn-m001-cray uas admin uais create --class-id d764c880-41b8-41e8-bacc-f94f7c5b053d --owner broker
      

      Example output:

      uai_age = "0m"
      uai_connect_string = "ssh broker@34.136.140.107"
      uai_host = "ncn-w003"
      uai_img = "registry.local/cray/cray-uai-broker:1.2.4"
      uai_ip = "34.136.140.107"
      uai_msg = ""
      uai_name = "uai-broker-f5bfb28c"
      uai_status = "Running: Ready"
      username = "broker"
      
      [uai_portmap]
      

Customize the Broker UAI SSH Configuration

The SSH configuration used on Broker UAIs resides in /etc/switchboard/sshd_config and contains the following:

Port 30123
AuthorizedKeysFile  .ssh/authorized_keys
UsePAM yes
X11Forwarding yes
Subsystem sftp  /usr/lib/ssh/sftp-server
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
AcceptEnv UAI_ONE_SHOT
UseDNS no

Match User !root,*
  PermitTTY yes
  ForceCommand /usr/bin/switchboard broker --class-id $UAI_CREATION_CLASS

The important content here is as follows:

  • Port 30123 tells sshd to listen on a port that can be reached through port forwarding by the publicly visible Kubernetes service.
  • The UseDNS no avoids any DNS issues resulting from the Broker UAI running in the Kubernetes network space.
  • The permitTTY yes setting permits interactive UAI logins.
  • The ForceCommand ... statement ensures that users are always sent on to End-User UAIs or drop out of the Broker UAI on failure, preventing users from directly accessing the Broker UAI.
  • The AcceptEnv UAI_ONE_SHOT setting is not required, but it allows a user to set the UAI_ONE_SHOT variable which instructs the broker to delete any created End-User UAI after the user logs out.

These should be left unchanged. The rest of the configuration can be customized as needed.

The following is an example that follows on from the previous section and configures SSH to provide a pre-login banner. Both a new banner file and a new sshd_config are placed in a Kubernetes ConfigMap and mounted over /etc/switchboard:

  1. Create a new pre-login banner file.

    NOTE A special “here document” form is used to prevent variable substitution in the file.

    ncn-m001-cat <<-"EOF" > banner
    Here is a banner that will be displayed before login on
    the Broker UAI
    
    EOF
    
  2. Create a new sshd_config.

    NOTE A special “here document” form is used to prevent variable substitution in the file.

    ncn-m001-cat <<-"EOF" > sshd_config
    Port 30123
    AuthorizedKeysFile  .ssh/authorized_keys
    UsePAM yes
    X11Forwarding yes
    Subsystem sftp  /usr/lib/ssh/sftp-server
    AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
    AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
    AcceptEnv LC_IDENTIFICATION LC_ALL
    AcceptEnv UAI_ONE_SHOT
    UseDNS no
    Banner /etc/switchboard/banner
    
    Match User !root,*
      PermitTTY yes
      ForceCommand /usr/bin/switchboard broker --class-id $UAI_CREATION_CLASS
    EOF
    
  3. Add the new banner file and sshd_config to a Kubernetes ConfigMap.

    ncn-m001-kubectl create configmap -n uas broker-sshd-conf --from-file sshd_config --from-file banner
    
  4. Mount the changes over /etc/switchboard.

    ncn-m001-cray uas admin config volumes create \
                --mount-path /etc/switchboard \
                --volume-description '{"config_map": {"name": "broker-sshd-conf", "default_mode": 384}}' \
                --volumename broker-sshd-config
    

    Example output:

    mount_path = "/etc/switchboard"
    volume_id = "4577eddf-d81e-40c9-9c91-082f3193edd6"
    volumename = "broker-sshd-config"
    
    [volume_description.config_map]
    default_mode = 384
    name = "broker-sshd-conf"
    
  5. Update the UAI class.

    ncn-m001-cray uas admin config classes update \
    --volume-list '4577eddf-d81e-40c9-9c91-082f3193edd6,11a4a22a-9644-4529-9434-d296eef2dc48,1ec36af0-d5b6-4ad9-b3e8-755729765d76,2246bbb1-4006-4b11-ba57-6588a7b7c02f,a3b149fd-c477-41f0-8f8d-bfcee87fdd0a' \
    d764c880-41b8-41e8-bacc-f94f7c5b053d --format yaml
    

    Example output:

    class_id: d764c880-41b8-41e8-bacc-f94f7c5b053d
    comment: UAI broker class
    default: false
    image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
    namespace: uas
    opt_ports: []
    priority_class_name: uai-priority
    public_ip: true
    replicas: 1
    resource_config:
    resource_id:
    service_account:
    timeout:
    tolerations:
    uai_compute_network: false
    uai_creation_class: bdb4988b-c061-48fa-a005-34f8571b88b4
    uai_image:
      default: false
      image_id: 8f180ddc-37e5-4ead-b261-2b401914a79f
      imagename: registry.local/cray/cray-uai-broker:1.2.4
    volume_list:
    - 4577eddf-d81e-40c9-9c91-082f3193edd6
    - 11a4a22a-9644-4529-9434-d296eef2dc48
    - 1ec36af0-d5b6-4ad9-b3e8-755729765d76
    - 2246bbb1-4006-4b11-ba57-6588a7b7c02f
    - a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
    volume_mounts:
    - mount_path: /etc/switchboard
      volume_description:
        config_map:
          default_mode: 384
          name: broker-sshd-conf
      volume_id: 4577eddf-d81e-40c9-9c91-082f3193edd6
      volumename: broker-sshd-config
    - mount_path: /etc/localtime
      volume_description:
        host_path:
          path: /etc/localtime
          type: FileOrCreate
      volume_id: 11a4a22a-9644-4529-9434-d296eef2dc48
      volumename: timezone
    - mount_path: /etc/sssd
      volume_description:
        secret:
          default_mode: 384
          secret_name: broker-sssd-conf
      volume_id: 1ec36af0-d5b6-4ad9-b3e8-755729765d76
      volumename: broker-sssd-config
    - mount_path: /app/broker
      volume_description:
        config_map:
          default_mode: 493
          name: broker-entrypoint
      volume_id: 2246bbb1-4006-4b11-ba57-6588a7b7c02f
      volumename: broker-entrypoint
    - mount_path: /lus
      volume_description:
        host_path:
          path: /lus
          type: DirectoryOrCreate
      volume_id: a3b149fd-c477-41f0-8f8d-bfcee87fdd0a
      volumename: lustre
    
  6. Once the new configuration is installed, clean out the old UAIs and restart the broker.

    NOTE Clearing out existing UAIs will terminate any user activity on those UAIs, make sure that users are warned of the disruption.

    1. Clean out the old UAIs.

      ncn-m001-cray uas admin uais delete --class-id bdb4988b-c061-48fa-a005-34f8571b88b4
      
      ncn-m001- cray uas admin uais delete --class-id d764c880-41b8-41e8-bacc-f94f7c5b053d
      

      Output similar to results = [ "Successfully deleted uai-vers-e937b810",] will be returned for each command.

    2. Restart the broker.

      ncn-m001-cray uas admin uais create --class-id d764c880-41b8-41e8-bacc-f94f7c5b053d --owner broker
      

      Example output:

      uai_age = "0m"
      uai_connect_string = "ssh broker@104.197.32.33"
      uai_host = "ncn-w003"
      uai_img = "registry.local/cray/cray-uai-broker:1.2.4"
      uai_ip = "104.197.32.33"
      uai_msg = "PodInitializing"
      uai_name = "uai-broker-ed144660"
      uai_status = "Waiting"
      username = "broker"
      
      [uai_portmap]
      
  7. Connect to the broker to log in:

    vers> ssh vers@104.197.32.33
    Here is a banner that will be displayed before login to SSH
    on Broker UAIs
    Password:
    

Top: User Access Service (UAS)

Next Topic: Customize End-User UAI Images