Working With aarch64 Images

Systems may have aarch64 (arm64) compute nodes present, but the Kubernetes nodes are always going to be running x86 hardware. This presents a challenge in creating and modifying compute images through IMS running on the Kubernetes cluster. To solve this, aarch64 hardware is emulated using Kata VMs and QEMU emulation software.

Kata and QEMU emulation

QEMU

QEMU is a generic open source emulator. qemu-user-static is inserted via binfmt_misc into the kernel as a translator. The aarch64 binaries are then recognized by the kernel and the correct emulation applied to run them on the x86 hardware.

More information can be found on the technologies here:

Kata

Because of the level of kernel interaction required by QEMU and the fact that most recipe builds and image customization jobs are running as the root user, this would open a fairly significant security hole if these Kubernetes pods were running directly on the worker nodes like normal pods.

In order to keep the system secure, the emulation pods are being run inside Kata VMs with a different running kernel than the worker node, so it is not possible through a kernel bug to get access to the running worker kernel. Each IMS job pod is run inside its own Kata VM so there is no possibility of breaking out from one job into another.

For more information on Kata, see Kata.

Performance

Because of the emulation and needing to run inside of a VM, the performance of the aarch64 image building and customization is much slower than the same operation being done for x86 images running on native hardware; there typically is around a 10 times slowdown in this configuration. This is unfortunate, but unavoidable given the need to work on the existing x86 management nodes.

Remote build nodes

Compute nodes may be converted into remote build nodes to speed up recipe builds and image customizations if there are aarch64 compute nodes available. The IMS job will be run on the compute node and utilizing native hardware it will perform much better than any emulated job.

For more information on setting up and using remote build nodes, see Configure a Remote Build Node.

Specifying architecture for recipes and images

For the most part, the architecture is handled automatically once a recipe is labeled with having an architecture of aarch64. If the recipe is being installed via a package, the architecture may be set in the manifest file. If the recipe is being installed manually, there is an option using the cray CLI to set or modify the architecture.

The architecture flag in the recipe or image will be used to set up the correct environment for the IMS job - either emulation or running on the native hardware. Once they are tagged, no further user input is required for the IMS jobs to run correctly.

When a recipe is built into an image, the resulting image is automatically set with the correct architecture picked up from the recipe. When an image is customized, the resulting image will have the same architecture as the original base image used for the customization. No manual changes are required in these workflows.

Importing a recipe

  1. (ncn-mw#) Create the new recipe record.

    When a new recipe is imported or created in IMS there is an architecture flag that will set that information into the recipe record.

        cray ims recipes create --name "My Recipe" \
            --recipe-type kiwi-ng --linux-distribution sles15 \
            --arch "aarch64" --require-dkms False --format toml
    

    Expected output will look something like:

    created = "2022-12-04T17:25:52.482514+00:00"
    id = "2233c82a-5081-4f67-bec4-4b59a60017a6"
    linux_distribution = "sles15"
    name = "My Recipe"
    recipe_type = "kiwi-ng"
    arch = "aarch64"
    require_dkms = false
    

Updating an existing recipe

If a recipe record is created with the incorrect architecture, that field can be updated.

  1. (ncn-mw#) Look at the existing recipe record.

    cray ims recipes describe $IMS_RECIPE_ID --format toml
    

    Expected output will look something like this:

    arch = "x86_64"
    created = "2023-06-26T19:18:50.618917+00:00"
    id = "da4121c2-2681-40f9-8007-4dcccf379e24"
    linux_distribution = "sles15"
    name = "cos-2.6.71-20230622190918-sles15sp5.aarch64"
    recipe_type = "kiwi-ng"
    require_dkms = false
    [[template_dictionary]]
    key = "COS_PRODUCT_VERSION"
    value = "2.6.71-20230622190918"
    
    [[template_dictionary]]
    key = "SHS_VERSION"
    value = "master"
    
    [link]
    etag = "38dcc9d03b8bf1fcd1bb4fd660607bc0"
    path = "s3://ims/recipes/da4121c2-2681-40f9-8007-4dcccf379e24/recipe.tar.gz"
    type = "s3"
    
  2. (ncn-mw#) If the architecture is wrong, update it.

    cray ims recipes update --arch aarch64 $IMS_RECIPE_ID --format toml
    

    Expected output will look something like:

    arch = "aarch64"
    created = "2023-06-26T19:18:50.618917+00:00"
    id = "da4121c2-2681-40f9-8007-4dcccf379e24"
    linux_distribution = "sles15"
    name = "cos-2.6.71-20230622190918-sles15sp5.aarch64"
    recipe_type = "kiwi-ng"
    require_dkms = false
    [[template_dictionary]]
    key = "COS_PRODUCT_VERSION"
    value = "2.6.71-20230622190918"
    
    [[template_dictionary]]
    key = "SHS_VERSION"
    value = "master"
    
    [link]
    etag = "38dcc9d03b8bf1fcd1bb4fd660607bc0"
    path = "s3://ims/recipes/da4121c2-2681-40f9-8007-4dcccf379e24/recipe.tar.gz"
    type = "s3"
    

Importing an image

  1. (ncn-mw#) Create the new image record.

    When a new image is imported or created in IMS there is an architecture flag that will set that information into the image record.

    cray ims images create --name "My New Image" --arch aarch64 --format toml
    

    Example output:

    id = "0a6459dc-aa54-432c-9013-3963a4d0f578"
    arch = "aarch64"
    created = "2023-06-28T21:46:50.181687+00:00"
    name = "My New Image"
    

Updating an existing image

If an image is uploaded into IMS with the incorrect architecture specified, that field can be updated.

  1. (ncn-mw#) Look at the image record.

    cray ims images describe $MY_IMS_IMAGE_ID --format json
    

    Expected output:

    {
        "arch": "x86_64",
        "created": "2023-06-27T01:59:26.116060+00:00",
        "id": "5c0a8f43-462c-414d-924a-b73ae404f4e0",
        "link": {
            "etag": "90e02dc3c17a8a0d8f1b2e90f34d76ea",
            "path": "s3://boot-images/5c0a8f43-462c-414d-924a-b73ae404f4e0/manifest.json",
            "type": "s3"
        },
        "name": "sp4-aarch64-2.6.71-image"
    }
    
  2. (ncn-mw#) If the architecture is incorrect, update it to the correct value.

    cray ims images update --arch aarch64 $MY_IMS_IMAGE_ID --format json
    

    Expected output:

    {
        "arch": "aarch64",
        "created": "2023-06-27T01:59:26.116060+00:00",
        "id": "5c0a8f43-462c-414d-924a-b73ae404f4e0",
        "link": {
            "etag": "90e02dc3c17a8a0d8f1b2e90f34d76ea",
            "path": "s3://boot-images/5c0a8f43-462c-414d-924a-b73ae404f4e0/manifest.json",
            "type": "s3"
        },
        "name": "sp4-aarch64-2.6.71-image"
    }