There is a known bug in Kubernetes 1.19.9 where movement of a pod with an attached volume may not complete in time and cause the kubelet service to stream error messages to the /var/log/messages log file. If this goes unchecked, it will fill up the root file system.
Log into the node that has space issues.
Verify that you have a large messages file in /var/log/.
ncn-m/w:/var/log # ls -lh messages-20211212
-rw-r----- 1 root root 67G Dec 13 12:24 messages-20211212
Remove the file.
Restart kubelet to address the streaming log entries.
ncn-m/systemctl restart kubelet.service
Restart the syslog service.
ncn-m/systemctl restart rsyslog
Verify that the space issue is resolved
ncn-m/df -h /
Filesystem Size Used Avail Use% Mounted on
LiveOS_rootfs 280G 933M 279G 1% /