Kubernetes can fail to detect that the main CFS Batcher process has crashed. This means that Kubernetes does not restart the pod, causing Batcher to not work until an administrator notices this and manually restarts it. In the meantime, CFS Automatic Configuration Management will not be operational.
There are two issues with CFS Batcher that lead to this problem:
At a high level, an administrator will first notice that no CFS sessions are being created by Batcher.
(ncn-mw#) Looking at the Kubernetes pod log can confirm that the main process has hit a fatal exception.
kubectl logs -n services -l app.kubernetes.io/instance=cray-cfs-batcher
Any fatal exception will cause the main problem (batcher being crashed and not restarting). The specific known exception above (when it fails to get a list of sessions from CFS) results in log messages like the following:
2023-09-19 06:44:29,956 - INFO - batcher.batch - Waiting for CFS to become available
2023-09-19 06:44:31,176 - ERROR - batcher.cfs.sessions - Unexpected response from CFS: 404 Client Error: Not Found for url:
http://cray-cfs-api/v3/sessions
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/app/lib/batcher/_main_.py", line 99, in <module>
main()
File "/app/lib/batcher/_main_.py", line 82, in main
manager = BatchManager()
File "/app/lib/batcher/batch.py", line 72, in _init_
self._rebuild_state()
File "/app/lib/batcher/batch.py", line 176, in _rebuild_state
for session in sessions.iter_sessions():
File "/app/lib/batcher/cfs/sessions.py", line 63, in iter_sessions
for session in data["sessions"]:
TypeError: 'NoneType' object is not subscriptable
(ncn-mw#) If this issue is encountered, then batcher can be brought back to life by killing the current pod.
kubectl delete pod -n services -l app.kubernetes.io/instance=cray-cfs-batcher
The fix addresses both the known crash, and the failure of the liveness checker to detect when the main process has crashed.