Kubernetes Master or Worker node’s root filesystem is out of space

Description

There is a known bug in Kubernetes 1.19.9 where movement of a pod with an attached volume may not complete in time and cause the kubelet service to stream error messages to the /var/log/messages log file. If this goes unchecked, it will fill up the root file system.

Fix

  1. Log into the node that has space issues.

  2. Verify that you have a large messages file in /var/log/.

    ncn-mw# cd /var/log
    ncn-mw# ls -lh messages-20211212
    -rw-r----- 1 root root 67G Dec 13 12:24 messages-20211212
    
  3. Remove the file.

  4. Restart kubelet to address the streaming log entries.

    ncn-mw# systemctl restart kubelet.service
    
  5. Restart the syslog service.

    ncn-mw# systemctl restart rsyslog
    
  6. Verify that the space issue is resolved

    ncn-mw# df -h /
    Filesystem      Size  Used Avail Use% Mounted on
    LiveOS_rootfs   280G  933M  279G   1% /