Troubleshoot Common Vault Cluster Issues

Search for underlying issues causing unhealthy Vault clusters. Check the Vault statefulset and various pod logs to determine what is impacting the health of the Vault.

Procedure

  1. View the Vault statefulset.

    ncn-mw# kubectl -n vault get statefulset --show-labels
    

    Example output:

    NAME         READY   AGE   LABELS
    cray-vault   3/3     8d    app.kubernetes.io/name=vault,vault_cr=cray-vault
    
  2. Check the pod logs for the bank-vaults container for Vault statefulset pods.

    ncn-mw# kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults
    ncn-mw# kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c bank-vaults
    ncn-mw# kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults
    
  3. Check the Vault container logs within the pod.

    ncn-mw# kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c vault
    ncn-mw# kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c vault
    ncn-mw# kubectl logs -n vault cray-vault-2 --tail=-1 --prefix -c vault
    
  4. Check the Vault operator pod logs using ephemerally named pods.

    ncn-mw# kubectl logs -n vault cray-vault-operator-7dbbdbb68b-zvg2g --tail=-1 --prefix