If there has been a failure to initialize all Ceph CSI components on ncn-s001, then the storage node cloud-init may need to be rerun.
Verify that the ceph-csi requirements are in place
Log in to ncn-s00 and run this command
ncn-s001# ceph -s
If it returns a connection error then assume Ceph is not installed. See Rerun Storage Node cloud-init.
Verify all post-Ceph-install tasks have run
Log in to ncn-s001 and check /etc/cray/ceph for completed task files (ceph_k8s_initialized & csi_initialized).
ncn-s001# ls /etc/cray/ceph/
ceph_k8s_initialized csi_initialized installed kubernetes_nodes.txt tuned
Check your results against this example.
If any components are missing, See Rerun Storage Node cloud-init.
Check to see if k8s ceph-csi prerequisites have been created
These commands can be run this from any master, worker node, or ncn-s001.
ncn# kubectl get cm
NAME DATA AGE
ceph-csi-config 1 3h50m
cephfs-csi-sc 1 3h50m
kube-csi-sc 1 3h50m
sma-csi-sc 1 3h50m
sts-rados-config 1 4h
ncn# kubectl get secrets | grep csi
csi-cephfs-secret Opaque 4 3h51m
csi-kube-secret Opaque 2 3h51m
csi-sma-secret Opaque 2 3h51m
Check your results against the above examples.
If any components are missing, see Rerun Storage Node cloud-init.
This procedure will restart the storage node cloud-init process to prepare Ceph for use by the utility storage nodes.
Run the following:
ncn-s001# ls /etc/cray/ceph
If any files are there they will represent completed stages.
If you have a running cluster you will want to edit ‘storage-ceph-cloudinit.sh` on ncn-s001
ncn-s001# vi /srv/cray/scripts/common/storage-ceph-cloudinit.sh
Comment out this section:
#if [ -f "$ceph_installed_file" ]; then
# echo "This ceph cluster has been initialized"
#else
# echo "Installing ceph"
# init
# mark_initialized $ceph_installed_file
#fi
Run the storage-ceph-cloudinit.sh script:
ncn-s001# /srv/cray/scripts/common/storage-ceph-cloudinit.sh
Configuring node auditing software
Using generic auditing configuration
This ceph cluster has been initialized
This ceph cluster has already been tuned
This ceph radosgw config and initial k8s integration already complete
ceph-csi configuration has been already been completed