I've noticed this error that appears in the celery logs:
==> /home/cchq/www/monolith/log/celery_flower.log <==
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/cchq/www/monolith/releases/2022-02-28_03.59/settings.py", line 1601, in <module>
SHARED_DRIVE_CONF = helper.SharedDriveConfiguration(
File "/home/cchq/www/monolith/releases/2022-02-28_03.59/settingshelper.py", line 27, in __init__
self.blob_dir = self._init_dir(blob_dir)
File "/home/cchq/www/monolith/releases/2022-02-28_03.59/settingshelper.py", line 35, in _init_dir
os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/shared_monolith/blobdb'
Appears to be related to running a django-manage check_services
I put a watch on ls /opt/shared_monolith -l and every time check_services runs (just before it completes it seems), the /opt/shared_monolith/blobdb symlink timestamp updates - is it being recreated during the check_services?
At the same time, the "FileExistsError: [Errno 17] File exists: '/opt/shared_monolith/blobdb" error appears in the /home/cchq/www/monolith/log/celery_flower.log
It doesn't, however, appear to be related to celery stalling.
EDIT that last statement may be incorrect, I'm making an assumption on the appearance of a new "FileExistsError" entry in celery_flower.log every time I run check_services. Let me double check that.
Yep, apologies, that error seems to have been a bit of a red herring. It's not appearing every time I do a django-manage check_services. The blobdb symlink timestamp is being updated though. Possibly normal behaviour. Celery has been up for 30 minutes or so now - the longest in a while. I'm going to try syncing with the server to see how it behaves.