Troubleshoot Ceph MDS Client Connectivity Issues

Use this procedure to diagnose and fix clients not logging into Ceph FS.

NOTE: This section does not diagnose nor fix network issues. Please ensure that all networking is functional before proceeding.

IMPORTANT: The following commands can be run from ncn-m001/2/3 or ncn-s001/2/3.

Procedure

  1. Identify if clients are not logged into Ceph FS.

    ceph fs status
    

    Example output:

    cephfs - 0 clients    <---- This indicates we have no clients connected
    ======
    RANK      STATE                MDS               ACTIVITY     DNS    INOS
    0        active      cephfs.ncn-s001.abiiiw  Reqs:    0 /s     0      0
    0-s   standby-replay  cephfs.ncn-s002.kyayma  Evts:   38 /s  35.5k  3220
         POOL         TYPE     USED  AVAIL
    cephfs_metadata  metadata  2403M  11.1T
    cephfs_data      data    2641G  11.1T
        STANDBY MDS
    cephfs.ncn-s003.sjatdm
    
  2. Fail over the MDS to trigger clients logins.

    ceph mds fail 0
    

    NOTE: “0” refers to the active rank in our above output.

  3. Verify clients have reconnected.

    ceph fs status
    

    Example output:

    cephfs - 24 clients   <---- Shows our clients have reconnected
    ======
    RANK      STATE                MDS               ACTIVITY     DNS    INOS
     0        active      cephfs.ncn-s002.kyayma  Reqs:    1 /s  52.8k  20.3k
    0-s   standby-replay  cephfs.ncn-s003.sjatdm  Evts:    0 /s     0      0
          POOL         TYPE     USED  AVAIL
    cephfs_metadata  metadata  2404M  11.1T
      cephfs_data      data    2641G  11.1T
         STANDBY MDS
    cephfs.ncn-s001.abiiiw
    MDS version: ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)