S3FS Usage and Guidelines for Shasta

Introduction

S3FS is a FUSE-based filesystem backed by Amazon S3. CSM uses it to provide temporary overflow storage, as well as to support SDU and NMD services by providing space for dumps.

When to use

Use S3FS for short term storage of large files or RPMs, to avoid filling up the root volume.

When NOT to use

  • For long term storage.
    • This is ONLY meant to provide temporary relief.
    • A vigilant practice of cleaning up unused files should be enforced.
    • This is particular important on master NCNs.
      • The SDU uses S3FS on master nodes.
      • The S3FS cache partition must have sufficient free space in order for SDU to function.
  • A place to uncompress tar files.
    • This puts unnecessary load on the storage cluster, because uncompressing a tar file requires a lot of reads and writes back to the object storage endpoints.
  • Running programs from the S3FS mount point.
    • This eats into memory for long running programs, and the programs may not perform properly.

Cache pruning

S3FS maintains a local cache, which can fill up if left unchecked, so CSM periodically clears the cache. The S3FS cache parent directory is /var/lib/s3fs_cache, with a subdirectory corresponding to a given S3FS mount point. On master nodes, this will contain an sds/ subdirectory. Otherwise, on all other nodes, it will contain a boot-images/ subdirectory.

Periodically, two cron jobs clear out cache files:

  • sds/ is cleared at 00:05 daily when it surpasses 100 GiB
  • boot-images/ is cleared at 00:00 daily when it surpasses 150 GiB

Note: Other arbitrary files and directories directly under /var/lib/s3fs_cache will not be pruned, and will continue to occupy disk space until they are deleted manually. Do not store large files in this directory expecting them to be pruned.

The contents of the S3FS cache can be safely deleted, because the cache is repopulated on demand.

Also see Troubleshoot S3FS Cache Cleanup.

Additional considerations

How to use

Gathering credentials from the Rados Gateway

  • Replace <radosgw-user> below with the UID for the Rados Gateway/S3 user ID.
  • Use a meaningful filename for storing the credentials.
    • Replace <filename> below with the credentials filename.
  • Create a mount location.
    • Replace <mount path> below with the mount location.
radosgw-admin user info --uid <radosgw-user>|jq -r '.keys[]|.access_key +":"+ .secret_key' >>${HOME}/.<filename>.s3fs && \
chmod 600 ~/.<filename>.s3fs && \
mkdir <mount path>

Mounting the volume

Mounting without cache

s3fs <radosgw-user> <mount path>  -o passwd_file=${HOME}/.<filename>.s3fs,url=http://rgw-vip.nmn,use_path_request_style

Mounting with cache

IMPORTANT: To use this option there must be a dedicated landing space that is a partition. This ensures that the usage does not impact the root drive.

s3fs <radosgw-user> <mount path>  -o passwd_file=${HOME}/.<filename>.s3fs,url=http://rgw-vip.nmn,use_path_request_style,use_cache=<dedicated_cache_partition_location>,check_cache_dir_exist=true

Unmounting the volume

umount <mount path>

Troubleshooting and additional information

The following links provide additional information about S3 and S3FS, including troubleshooting guides.