This procedure describes how to manually clean up S3FS cache when the automatic pruning is insufficient and the cache continues to grow, consuming excessive disk space on worker nodes.
For more information on S3FS cache pruning, see Cache pruning.
(ncn-w#) CSM includes an automatic S3FS cache pruning mechanism in the form of a daily cron job:
cat /etc/cron.d/prune-s3fs-boot-images-cache
Output:
0 0 * * * root /usr/bin/prune-s3fs-cache.sh boot-images /var/lib/s3fs_cache 161061273600 -silent
However, some files may not be properly pruned by the automated process. Over time, this may lead to increasing disk usage, which requires manual intervention.
/var/lib/s3fs_cache/ subdirectories(ncn-w#) Example of problematic cache growth:
cd /var/lib/s3fs_cache/ && \
ls -la && \
echo "Disk Usage:" && \
du -sh boot-images/ .boot-images.mirror/
Example output:
total 36
drwxr-xr-x 6 root root 4096 Nov 29 08:02 .
drwxr-xr-x 1 root root 4096 Nov 29 08:16 ..
drwxr-xr-x 39 root root 4096 Mar 5 11:40 boot-images
drwxr-xr-x 2 root root 4096 Mar 12 11:56 .boot-images.mirror
drwxr-xr-x 39 root root 4096 Mar 5 11:40 .boot-images.stat
drwx------ 2 root root 16384 Nov 29 08:43 lost+found
Disk Usage:
67G boot-images/
42G .boot-images.mirror/
The contents of the S3FS cache can be safely deleted, because the cache is repopulated on demand.
(ncn-w#) Perform the following steps on each affected worker node:
Check current disk usage.
df -h /var/lib/s3fs_cache/
du -sh /var/lib/s3fs_cache/*
Clean up files older than 30 days in the main cache directory.
find /var/lib/s3fs_cache/boot-images/ -atime +30 -type f | xargs rm -vf
Clean up files in the mirror directory.
find /var/lib/s3fs_cache/.boot-images.mirror/ -atime +30 -type f | xargs rm -vf
Clean up files in the stat directory.
find /var/lib/s3fs_cache/.boot-images.stat/ -atime +30 -type f | xargs rm -vf
Remove empty directories.
find /var/lib/s3fs_cache/ -type d -empty ! -wholename /var/lib/s3fs_cache/ -delete
Check the disk usage after cleanup.
df -h /var/lib/s3fs_cache/
In order to clean up files older than a different time period, adjust the -atime parameter.
For example:
(ncn-w#) Clean up files older than 7 days:
find /var/lib/s3fs_cache/boot-images/ -atime +7 -type f | xargs rm -vf
(ncn-w#) Clean up files older than 1 day:
find /var/lib/s3fs_cache/boot-images/ -atime +1 -type f | xargs rm -vf
(ncn-w#) If the cache is severely corrupted or wishing to start fresh, then perform
a complete cache reset.
Warning: This will remove all cached data, which may cause a temporary performance impact as the cache rebuilds.
cd /var/lib/s3fs_cache/
rm -rf boot-images/ .boot-images.mirror/ .boot-images.stat/