A deployment of HashiCorp Vault, managed via the Bitnami Bank-vaults operator, stores private and public Certificate Authority
(CA) material, and serves APIs through a PKI engine instance. This instance also serves as a general secrets engine for the system.
Kubernetes service account authorization is utilized to authenticate access to Vault. The configuration of Vault, as deployed on the system, can be viewed with the following command:
ncn-mw# kubectl get vault -n vault cray-vault -o yaml
A Kubernetes operator manages the deployment of Vault, based on this definition. The operator is deployed to the vault namespace.
The resulting instance is deployed to the vault namespace.
IMPORTANT: Changing the cray-vault custom resource definition or modifying data directly in Vault is not supported unless directed by customer support.
For more information, refer to the following resources:
In previous releases, Vault used etcd as a high-availability (HA) storage back-end. Currently, Vault uses HashiCorp’s Raft implementation. Raft is now configured to run natively inside the Vault StatefulSet instead of as an independent deployment.
Vault requires unseal keys for start-up. If the unseal keys are not present, or are incorrect, Vault (by design) will not start.
Unseal keys are stored in the cray-vault-unseal-keys Kubernetes Secret on a system, which is inside the vault namespace.
Administrative access to Vault can be accomplished through the use of the unseal secret. The use of administrative access should be limited to situations where it is truly necessary. Otherwise, Kubernetes service account access should be used.
To obtain and use the root token:
ncn-mw# VAULT_TOKEN=$(kubectl get secrets cray-vault-unseal-keys -n vault -o jsonpath={.data.vault-root} | base64 -d)
ncn-mw# kubectl exec -it -n vault -c vault cray-vault-0 -- sh -c "VAULT_ADDR=http://localhost:8200 VAULT_TOKEN=$VAULT_TOKEN vault secrets list"
Vault is configured to allow service account access from the services namespace (among others). This access is tied to a role,
which is also subject to specific access policies.
To obtain and use the service account token:
ncn-mw# SA_SECRET=$(kubectl -n services get serviceaccounts default -o jsonpath='{.secrets[0].name}')
ncn-mw# SA_JWT=$(kubectl -n services get secret $SA_SECRET -o jsonpath='{.data.token}' | base64 --decode)
ncn-mw# VAULT_TOKEN=$(kubectl exec -it -n vault -c vault cray-vault-0 -- sh -c \
"export VAULT_ADDR=http://localhost:8200; vault write auth/kubernetes/login role=services jwt=$SA_JWT -format=json" \
| jq ".auth.client_token" | sed -e 's/"//g')
ncn-mw# kubectl exec -it -n vault -c vault cray-vault-0 -- sh -c "VAULT_ADDR=http://localhost:8200 VAULT_TOKEN=$VAULT_TOKEN vault kv list secret/"
Service account tokens will eventually expire.
Check the status of Vault clusters with the following command:
ncn-mw# for n in $(seq 0 2); do echo "======= Vault status from cray-vault-${n} ======"; kubectl exec -it -n vault -c vault cray-vault-${n} -- sh -c "VAULT_ADDR=http://localhost:8200 vault status"; done
Example output:
======= Vault status from cray-vault-0 ======
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.5.5
Cluster Name vault-cluster-e19b13b8
Cluster ID 3ea3b6a2-f3f8-fda3-d997-454795dc2be5
HA Enabled true
HA Cluster https://cray-vault-1:8201
HA Mode standby
Active Node Address http://cray-vault.vault:8200
Raft Committed Index 521
Raft Applied Index 521
======= Vault status from cray-vault-1 ======
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.5.5
Cluster Name vault-cluster-e19b13b8
Cluster ID 3ea3b6a2-f3f8-fda3-d997-454795dc2be5
HA Enabled true
HA Cluster https://cray-vault-1:8201
HA Mode active
Raft Committed Index 521
Raft Applied Index 521
======= Vault status from cray-vault-2 ======
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.5.5
Cluster Name vault-cluster-e19b13b8
Cluster ID 3ea3b6a2-f3f8-fda3-d997-454795dc2be5
HA Enabled true
HA Cluster https://cray-vault-1:8201
HA Mode standby
Active Node Address http://cray-vault.vault:8200
Raft Committed Index 521
Raft Applied Index 521
Healthy clusters will have one Vault pod in active HA mode, and two Vault pods in standby HA mode. All instances should also be unsealed and initialized.