Best Practices To Follow To Avoid Issues With iSCSI SBPS

This document presents recommended best practices to avoid issues with iSCSI SBPS.

Scenario #1: As roots/PE images will be placed in s3 storage/IMS, there could be unused images which user/admin would like to delete. The following are steps to delete unused rootfs/PE images safely.

Steps to remove unused rootfs/PE images projected by iSCSI SBPS

  1. (ncn-w#) Identify the unused PE and rootfs images from targetcli ls command output from one of the iSCSI target node (worker node) and list them in a file.

    Example command:

    targetcli ls

    Example command output snippet:

    ...

|     | o- lun0  [fileio/a50dd52157e1636 (/var/lib/cps-local/boot-images/PE/CPE-amd.x86_64-23.12.squashfs) (default_tg_pt_gp)]
...
|     | o- lun5  [fileio/c1d98cf92b0647f (/var/lib/cps-local/boot-images/PE/CPE-aocc.x86_64-23.12.squashfs) (default_tg_pt_gp)]
...
|     | o- lun28  [fileio/84c5b467e892c6b (/var/lib/cps-local/boot-images/d114469b-06c6-42ed-b151-05a7f555b3a1/rootfs) (default_tg_pt_gp)]
...

  2. (ncn-w#) Provide a list of images in a file that are identified for deletion.

    Example:

    cat img_list

    CPE-amd.x86_64-23.12.squashfs
CPE-aocc.x86_64-23.12.squashfs
d114469b-06c6-42ed-b151-05a7f555b3a1

  3. (ncn-w#) Run the script get_img_str.sh on one of the worker node (For example, iSCSI target) which takes above file having list of images as an argument.

    Example command:

    sh get_img_str.sh img_list

    This script will create a output file named img_str.txt having list of image identifier strings for which corresponding iSCSI LUNs on the iSCSI initiator are to be deleted.

    Example img_str.txt file contents:

    a50dd52157e1636
c1d98cf92b0647f
84c5b467e892c6b

    The first image identifier string above corresponds to the image CPE-amd.x86_64-23.12.squashfs in the list and corresponds to lun0 in the targetcli ls output as below:

    |     | o- lun0  [fileio/a50dd52157e1636 (/var/lib/cps-local/boot-images/PE/CPE-amd.x86_64-23.12.squashfs) (default_tg_t_gp)]

    Similarly, the second (c1d98cf92b0647f) and third (84c5b467e892c6b) image strings correspond to lun5 and lun28 respectively.

  4. Login to iSCSI initiator node (compute/UAN) and check the luns corresponding to the image identifiers in img_str.txt.

    (nid00000# or uan0#) Example commands and their outputs:

    lsscsi | grep a50dd52157e1636

    [14:0:0:0]   disk    LIO-ORG  a50dd52157e1636  4.0   /dev/sdg
[15:0:0:0]   disk    LIO-ORG  a50dd52157e1636  4.0   /dev/sdao
[16:0:0:0]   disk    LIO-ORG  a50dd52157e1636  4.0   /dev/sdbw
[17:0:0:0]   disk    LIO-ORG  a50dd52157e1636  4.0   /dev/sdde

    lsscsi | grep c1d98cf92b0647f

    [14:0:0:5]   disk    LIO-ORG  c1d98cf92b0647f  4.0   /dev/sdl
[15:0:0:5]   disk    LIO-ORG  c1d98cf92b0647f  4.0   /dev/sdat
[16:0:0:5]   disk    LIO-ORG  c1d98cf92b0647f  4.0   /dev/sdcz
[17:0:0:5]   disk    LIO-ORG  c1d98cf92b0647f  4.0   /dev/sddj

    lsscsi | grep 84c5b467e892c6b

    [14:0:0:28]  disk    LIO-ORG  84c5b467e892c6b  4.0   /dev/sdai
[15:0:0:28]  disk    LIO-ORG  84c5b467e892c6b  4.0   /dev/sdbp
[16:0:0:28]  disk    LIO-ORG  84c5b467e892c6b  4.0   /dev/sdcc
[17:0:0:28]  disk    LIO-ORG  84c5b467e892c6b  4.0   /dev/sdeg

  5. (nid00000# or uan0#) Copy img_str.txt and rm_iscsi_luns.sh onto all iSCSI initiator nodes (compute/UAN nodes) and run the rm_iscsi_lun.sh script on compute and UAN nodes with img_str.txt as an argument.

    sh rm_iscsi_luns.sh img_str.txt

    (ncn-m#) This can also be run on multiple iSCSI initiator nodes using pdsh command as follows. The following command runs the script on four compute nodes: nid000001, nid000002, nid000003, and nid000004:

    pdsh -w nid00000[1-4]-nmn "sh rm_iscsi_luns.sh img_str.txt"

  6. (nid00000# or uan0#) Verify that the luns corresponding to the image identifiers are deleted from iSCSI initiator nodes (compute/UAN).

    lsscsi | grep -E 'a50dd52157e1636|c1d98cf92b0647f|84c5b467e892c6b'

  7. (ncn-mw#) Login to master node and delete the images listed in the img_list file.

    Example command:

    cray artifacts delete boot-images PE/CPE-amd.x86_64-23.12.squashfs

  8. Wait for 180 seconds and verify the iSCSI luns corresponding to the images are deleted by checking the targetcli ls output.

    targetcli ls | grep a50dd52157e1636

    This should not list any iSCSI luns or fileio backing store corresponding to the image identifier strings.