Healthchecks

The health of a the Shared File Systems API service can be determined with the help of a “healthcheck” middleware. This middleware is enabled by default with the api-paste file that is packaged with the software. There is hence a /healthcheck endpoint that responds to GET requests with HTTP 200 “OK” as the body if the API service is functional. If the API service is not functional, the response is HTTP 503 “Service Unavailable”.

This /healthcheck endpoint can be polled by load balancers to determine service availability. The end point behaves very similar to mod_status in apache. A sample configuration that can be added to the api-paste file is as follows.

[app:healthcheck]
paste.app_factory = oslo_middleware:Healthcheck.app_factory
backends = disable_by_file
disable_by_file_path = /etc/manila/healthcheck_disable
detailed = False

Example healthcheck requests and responses:

 $ curl -i -X GET http://203.0.113.30/share/healthcheck
 HTTP/1.1 200 OK
 Date: Wed, 20 Mar 2024 23:00:19 GMT
 Server: Apache/2.4.52 (Ubuntu)
 Content-Type: text/plain; charset=UTF-8
 Content-Length: 2
 Connection: close
 Vary: Accept-Encoding

 OK

 $ curl -i -X GET http://203.0.113.30/share/healthcheck -H "Accept: application/json"
 HTTP/1.1 200 OK
 Date: Wed, 20 Mar 2024 23:01:08 GMT
 Server: Apache/2.4.52 (Ubuntu)
 Content-Type: application/json
 Content-Length: 62
 Connection: close

 {
     "detailed": false,
     "reasons": [
         "OK"
     ]
 }

$ curl -i -X GET http://203.0.113.30/share/healthcheck -H "Accept: text/html"
 HTTP/1.1 200 OK
 Date: Wed, 20 Mar 2024 23:02:27 GMT
 Server: Apache/2.4.52 (Ubuntu)
 Content-Type: text/html; charset=UTF-8
 Content-Length: 239
 Connection: close
 Vary: Accept-Encoding

 <HTML>
 <HEAD><TITLE>Healthcheck Status</TITLE></HEAD>
 <BODY>

 <H2>Result of 1 checks:</H2>
 <TABLE bgcolor="#ffffff" border="1">
 <TBODY>
 <TR>

 <TH>
 Reason
 </TH>
 </TR>
 <TR>

     <TD>OK</TD>

 </TR>
 </TBODY>
 </TABLE>
 <HR></HR>

 </BODY>
 </HTML>

A “detail” response can be sought if detailed is set to True in the [app:healthcheck] section of the api paste configuration file. This is not done by default.

$ curl -i -X GET http://203.0.113.30/share/healthcheck -H "Accept: application/json"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2024 23:06:19 GMT
Server: Apache/2.4.52 (Ubuntu)
Content-Type: application/json
Content-Length: 4177
Connection: close

{
    "detailed": true,
    "gc": {
        "counts": [
            400,
            5,
            0
        ],
        "threshold": [
            700,
            10,
            10
        ]
    },
    "greenthreads": [
        "  File \"/opt/stack/data/venv/lib/python3.10/site-packages/paste/urlmap.py\", line 216, in __call__\n    return app(environ, start_response)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 129, in __call__\n    resp = self.call_func(req, *args, **kw)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 193, in call_func\n    return self.func(req, *args, **kwargs)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py\", line 121, in __call__\n    response = self.process_request(req)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 146, in __call__\n    return self.call_func(req, *args, **kw)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 193, in call_func\n    return self.func(req, *args, **kwargs)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 582, in process_request\n    body, content_type = functor(results, healthy)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 510, in _make_json_response\n    body['greenthreads'] = self._get_greenstacks()\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 464, in _get_greenstacks\n    traceback.print_stack(gt.gr_frame, file=buf)\n"
    ],
    "now": "2024-03-20 23:06:19.907279",
    "platform": "Linux-5.15.0-91-generic-x86_64-with-glibc2.35",
    "python_version": "3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]",
    "reasons": [
        {
            "class": "HealthcheckResult",
            "details": "Path '/etc/manila/healthcheck_disable' was not found",
            "reason": "OK"
        }
    ],
    "threads": [
        "  File \"/usr/lib/python3.10/threading.py\", line 973, in _bootstrap\n    self._bootstrap_inner()\n  File \"/usr/lib/python3.10/threading.py\", line 1016, in _bootstrap_inner\n    self.run()\n  File \"/usr/lib/python3.10/threading.py\", line 953, in run\n    self._target(*self._args, **self._kwargs)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/tooz/coordination.py\", line 208, in _beat_forever_until_stopped\n    self._dead.wait(has_to_sleep_for / 2.0)\n  File \"/usr/lib/python3.10/threading.py\", line 607, in wait\n    signaled = self._cond.wait(timeout)\n  File \"/usr/lib/python3.10/threading.py\", line 324, in wait\n    gotit = waiter.acquire(True, timeout)\n",
        "  File \"/opt/stack/data/venv/lib/python3.10/site-packages/paste/urlmap.py\", line 216, in __call__\n    return app(environ, start_response)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 129, in __call__\n    resp = self.call_func(req, *args, **kw)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 193, in call_func\n    return self.func(req, *args, **kwargs)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py\", line 121, in __call__\n    response = self.process_request(req)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 146, in __call__\n    return self.call_func(req, *args, **kw)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py\", line 193, in call_func\n    return self.func(req, *args, **kwargs)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 582, in process_request\n    body, content_type = functor(results, healthy)\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 511, in _make_json_response\n    body['threads'] = self._get_threadstacks()\n  File \"/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/healthcheck/__init__.py\", line 452, in _get_threadstacks\n    traceback.print_stack(stack, file=buf)\n"
    ]
}

You may disable the healthcheck endpoint dynamically by creating a file called /etc/manila/healthcheck_disable. The name of this file can be customized with the configuration option disable_by_file_path in the [app:healthcheck] section of the api paste configuration file.