Known Issue: admin-client-auth Not Found

Running the Install CSM Services script, the following error may occur:

ERROR   Step: Set Management NCNs to use Unbound --- Checking Precondition
+ Getting admin-client-auth secret
Error from server (NotFound): secrets "admin-client-auth" not found
+ Obtaining access token

Fix

This can occur if the keycloak-users-localize pod has not completed, and that can be caused by an intermittent Istio issue. Remediate the issue with the following procedure:

  1. Follow Troubleshoot Intermittent HTTP 503 Code Failures to verify that Istio is healthy.

  2. (ncn-mw#) Ensure that the keycloak-wait-for-postgres-* pod is in a Completed state.

    kubectl get po -n services | grep keycloak-wait-for-postgres

    Example output:

    keycloak-wait-for-postgres-1-pv85m                                0/2     Completed   0          15d

  3. (ncn-mw#) If the keycloak-wait-for-postgres-* pod is not in a Completed state, then resubmit the job.

    kubectl get job -n services -l app.kubernetes.io/name=keycloak-wait-for-postgres -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

    Example output:

    job.batch "keycloak-wait-for-postgres-1" deleted
job.batch/keycloak-wait-for-postgres-1 replaced

Once the keycloak-wait-for-postgres-* pod has completed, the keycloak-users-localize job should create the admin-client-auth secret and complete. At that point, resume the CSM install by re-running the failed command.