Node Management Workflows

The following workflows are high-level overviews of node management tasks. These workflows depict how services interact with each other during node management, and help to provide a deeper understanding of how the system functions.

The workflows and procedures in this section include:

Components involved in adding or removing nodes

Both of these workflows are based on the interaction of the System Layout Service (SLS) with other hardware management services (e.g. HSM, HBTD, HMNFD, MEDS).

Mentioned in these workflows:

  • System Layout Service (SLS) serves as a “single source of truth” for the system design. It details the physical locations of network hardware, compute nodes and cabinets. Further, it stores information about the network, such as which port on which switch should be connected to each compute node.
  • Hardware State Manager (HSM) monitors and interrogates hardware components in an HPE Cray EX system, tracking hardware state and inventory information, and making it available via REST queries and message bus events when changes occur.
  • HMS Notification Fanout Daemon (HMNFD) receives component state change notifications from the HSM. It fans notifications out to subscribers (typically compute nodes).
  • Endpoint Discovery Service (REDS/MEDS) manages initial discovery, configuration, and geolocation of Redfish-enabled BMCs. It periodically makes Redfish requests to determine if hardware is present or missing.
  • Heartbeat Tracker Service (HBTD) listens for heartbeats from components (mainly compute nodes). It tracks changes in heartbeats and conveys changes to the HSM.

Add nodes

Use cases for adding nodes

Administrator permanently adds select compute nodes to expand the system.

Components involved in adding nodes

See Components involved in adding or removing nodes.

Workflow overview for adding nodes

Add Node Workflow

The following sequence of steps occur during this workflow.

  1. Administrator updates SLS.

    Administrator creates a new hardware entry for the select component names (xnames) in SLS. Administrator enters the node component names (xnames) in the SLS input file.

  2. Administrator adds compute nodes.

    The administrator physically adds select compute nodes and powers them on. Because the nodes are unknown, the DHCP and TFTP servers give them the special initialization RAM disk. The compute nodes performs local configuration.

    The remaining steps occur automatically as different APIs interact with each other.

  3. Set BMC credentials.

    The compute node requests per-node BMC credentials. This message must include the MAC address of the BMC. A new set of credentials is generated by the discovery service.

    Once the compute node is powered on, initialized, and discovered, REDS gets details about the new node like IP address, MAC address, the username and password for a BMC, state etc.

  4. REDS/MEDS to SLS.

    REDS/MEDS query SLS database for information about the new node.

    For example: “What xname is connected to port XX on switch Y?”

  5. SLS to REDS/MEDS.

    SLS updates the discovery service with the new compute node and its xname.

    For example: “xname x0c0... is connected to port XX”.

  6. REDS/MEDS to HSM.

    Discovery services update HSM about the new Redfish endpoint for the node (e.g. the xname and IP address of the new node).

    For example: “x0c0... at IP address AAA.BBB.CCC.DDD”

  7. HSM to SLS.

    HSM queries SLS for NID and role assignments for the new node.

  8. SLS to HSM.

    HSM updates the node map based on information received from SLS.

  9. Node to HBTD.

    The Heartbeat Tracker Service receives heartbeats from the new compute node after the node is powered on.

  10. HBTD to HSM.

    The Heartbeat Tracker Service report the heartbeat status to HSM.

  11. HSM to HMNFD.

    HSM sends the new compute node state information with State as ON to HMNFD. HMNFD fans out these notifications to the subscribing compute nodes.

Remove nodes

Use cases for removing nodes

Administrator permanently removes select compute nodes to contract the system.

Components involved in removing nodes

See Components involved in adding or removing nodes.

Workflow overview for removing nodes

Remove Node Workflows

The following sequence of steps occur during this workflow.

  1. Administrator updates SLS.

    Administrator deletes the node entries with the specific component name (xname) from SLS. Note that if deleting a parent object, then the children are also deleted from SLS. If the child object happens to be a parent, then the deletion can cascade down levels. If deleting a child object, it does not affect the parent.

  2. Administrator physically removes the compute nodes.

    The administrator powers off and physically removes the compute nodes.

  3. No heartbeats in HBTD.

    The Heartbeat Tracker Service stops receiving heartbeats and updates the node status, first as Standby and then Off, as per Redfish event.

    Standby status implies that the node is no longer ready and presumed dead. It typically means that the heartbeat is lost. Off status implies that the location is not populated with a component.

  4. HBTD to HSM.

    The Heartbeat Tracker Service reports the heartbeat status to the HSM.

  5. REDS/MEDS detect no BMC.

    The discovery service detects that the BMC is not there.

  6. REDS/MEDS to SLS.

    REDS/MEDS query SLS database for information about the missing BMCs.

  7. SLS to REDS/MEDS.

    SLS updates the discovery service that the BMC was removed.

  8. REDS/MEDS to HSM.

    Discovery services update the HSM that the BMC Redfish endpoints for the nodes were removed. HSM marks the state of BMCs and the nodes as Empty.

    Empty state implies that the location is not populated with a component.

  9. HSM to HMNFD.

    HSM sends the compute node state information with State as Empty to HMNFD. HMNFD fans out this notification to the subscribing compute nodes.

Replace nodes

Move nodes