On one of our servers the queue appears to have stalled and needed a restart. The results of /admin/system/check_services produces this:
FAILURE (Took 0.02s) celery : analytics_queue is delayed for 1:00:28.244477 (max allowed is 0:30:00)
async_restore_queue has been blocked for 0:14:46.044439 (max allowed is 0:01:00)
background_queue has been blocked for 0:14:46.054863 (max allowed is 0:10:00)
case_import_queue has been blocked for 0:14:46.085465 (max allowed is 0:01:00)
case_rule_queue is delayed for 1:00:28.205859 (max allowed is 1:00:00)
celery has been blocked for 0:14:46.066732 (max allowed is 0:01:00)
celery_periodic has been blocked for 0:14:53.330107 (max allowed is 0:10:00)
email_queue has been blocked for 0:14:53.321611 (max allowed is 0:00:30)
export_download_queue has been blocked for 0:14:53.328948 (max allowed is 0:00:30)
repeat_record_queue is delayed for 1:00:28.188084 (max allowed is 1:00:00)
ucr_queue is delayed for 14:33:51.099824 (max allowed is 1:00:00)
I checked it less than 10 minutes ago and it reported OK for celery.
What does that message actually mean?
/serverup.txt, /serverup.txt?heartbeat and /serverup.txt?only=celery all report success and sudo supervisorctl status reports all workers are running
What can I assume from check_services under the circumstances? Should I be restarting Celery?
Also, what is the difference between /serverup.txt, /serverup.txt?heartbeat and /serverup.txt?only=celery
EDIT
So the reason for the blocked queue appears to be the corehq.apps.data_analytics.tasks.build_last_month_MALT task.
Once revoked, the system appears to be back to normal.
Is there a log I can look at to see what could have been the cause of the holdup?
Thanks!