S3FS Usage and Guidelines for Shasta

Introduction

S3FS is a near POSIX filesystem. CSM uses it to provide temporary overflow storage, as well as to support SDU and NMD services by providing space for dumps.

When to use

Use S3FS for short term storage of large files or RPMs, to avoid filling up the root volume.

When NOT to use

  • For long term storage.
    • This is ONLY meant to provide temporary relief.
    • A vigilant practice of cleaning up unused files should be enforced.
    • This is particular important on master NCNs.
      • The SDU uses S3FS on master nodes.
      • The S3FS cache partition must have sufficient free space in order for SDU to function.
  • A place to uncompress tar files.
    • This puts unnecessary load on the storage cluster, because uncompressing a tar file requires a lot of reads and writes back to the object storage endpoints.
  • Running programs from the S3FS mount point.
    • This eats into memory for long running programs, and the programs may not perform properly.

Additional considerations

How to use

Gathering credentials from the Rados Gateway

  • Replace <radosgw-user> below with the UID for the Rados Gateway/S3 user ID.
  • Use a meaningful filename for storing the credentials.
    • Replace <filename> below with the credentials filename.
  • Create a mount location.
    • Replace <mount path> below with the mount location.
radosgw-admin user info --uid <radosgw-user>|jq -r '.keys[]|.access_key +":"+ .secret_key' >>${HOME}/.<filename>.s3fs && \
chmod 600 ~/.<filename>.s3fs && \
mkdir <mount path>

Mounting the volume

Mounting without cache

s3fs <radosgw-user> <mount path>  -o passwd_file=${HOME}/.<filename>.s3fs,url=http://rgw-vip.nmn,use_path_request_style

Mounting with cache

IMPORTANT: To use this option there must be a dedicated landing space that is a partition. This ensures that the usage does not impact the root drive.

s3fs <radosgw-user> <mount path>  -o passwd_file=${HOME}/.<filename>.s3fs,url=http://rgw-vip.nmn,use_path_request_style,use_cache=<dedicated_cache_partition_location>,check_cache_dir_exist=true

Unmounting the volume

umount <mount path>

Troubleshooting and additional information

The following links provide additional information about S3 and S3FS, including troubleshooting guides.