Paging CFS Records

NOTE: Paging is only available using the v3 CFS API. The v2 API will only return an error if the response size is too large.

When listing CFS configurations, sessions, or components, CFS lists only a limited number of records at a time. This helps reduce the memory requirements for both the CFS API and for the client, especially on systems with large numbers of components.

By default, the maximum number of records that CFS will return in a single query is specified by the Default page size CFS Global Option. However, this can be overridden at query time with the limit parameters. Pages beyond the first can be requested using the after_id parameter, which should be set to the id of the last record in the previous page. For convenience, each response includes a next section, which includes the after_id that should be used to request the next page, as well as the value of all other parameters needed to ensure consistent paging.

Paging with the CLI

  • (ncn-mw#) Limit the number of records returned in a query by using the --limit parameter.

    cray cfs v3 components list --limit 1 --format json

    Example response:

    {
  "components": [
    {
      "configuration_status": "configured",
      "desired_config": "example-config",
      "enabled": true,
      "error_count": 0,
      "id": "x3000c0s1b0n0",
      "logs": "ara.cmn.site/hosts?name=x3000c0s11b0n0",
      "tags": {}
    }
  ],
  "next": {
    "after_id": "x3000c0s1b0n0",
    "config_details": false,
    "config_name": "",
    "enabled": null,
    "ids": "",
    "limit": 1,
    "state_details": false,
    "status": "",
    "tags": ""
  }
}

  • (ncn-mw#) Request the next set of records using the --after-id parameter.

    cray cfs v3 components list --limit 1 --after-id x3000c0s1b0n0

Changing the default page size

(ncn-mw#) The default page size can be updated. For example:

cray cfs v3 options update --default-page-size 500

Use caution when changing this value, because the CFS services will use this value internally. If this is set too high on large systems, then the CFS API may struggle unless the memory limits for the CFS API pods are also increased. This option also controls the size at which the v2 API will throw an error when the response size is too large, so changing this can also impact the v2 API.

Programmatic iteration

All responses to CFS queries listing records include a next section, which contains the values needed to query the next page. When no more records are available and the last page is queried, the next value is empty. This can be used to easily iterate over multiple pages. This is shown in the example Python code below, where the next section is used directly as query parameters for the next query:

def iter_components(**kwargs):
    next_parameters = kwargs
    while True:
        response = requests.get(ENDPOINT, params=next_parameters)
        data = response.json()
        for component in data["components"]:
            yield component
        next_parameters = data["next"]
        if not next_parameters:
            break

Known issues

The addition of paging caused problems for some services and products in CSM 1.5.0. For more information, see CFS V2 Failures On Large Systems.