CFS Component With Zero-Length ID

In CSM versions lower than 1.7, it is possible in some situations for a CFS component to be created with a zero-length string for an id field (either manually by an administrator, or by the CFS Hardware Synchronization Agent). In CSM 1.7, the bug that allowed these to be created has been fixed, but such a component could still exist on systems that were upgraded to CSM 1.7. Such a component cannot be deleted using the Cray CLI or the CFS API.

Symptoms

An administrator may notice the invalid component when listing components in CFS. The presence of such a component could cause problems in different ways:

  • It may also cause the cmsdev test to fail. For details on these test failure symptoms, see Invalid CFS component.
  • Any tooling which parses CFS component listings may fail in unexpected ways when confronted with the zero-length ID string.

Remediation

Follow this procedure to remove the invalid CFS component.

  1. (ncn-mw#) Identify a running CFS server Kubernetes pod.

    POD=$(kubectl get pods -n services -l 'app.kubernetes.io/instance=cray-cfs-api' --no-headers | grep -w Running | awk '{ print $1 }' | head -1)
echo "${POD}"

    Example output:

    cray-cfs-api-7dfb78c7b8-clrjq

  2. (ncn-mw#) Delete the CFS component with the zero-length ID.

    No output is expected from the following command.

    kubectl exec -it -n services "${POD}" -- python3 -c 'from cray.cfs.api.controllers.components import delete_component_v2 ; delete_component_v2("")'

  3. (ncn-mw#) Verify that the invalid component has been removed from CFS. If this problem was detected by cmsdev, then run the following command to perform only the CFS subtest.

    /usr/local/bin/cmsdev test -q cfs