Celery reporting down, but appears up?

Developers
1

After a server reboot and check_services, celery reports it's been down for over 18 hours:

(cchq) ccc@monolith:~/commcare-cloud$ cchq monolith django-manage check_services
ssh ccc@10.3.0.4 -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.3 LTS
SUCCESS (Took   0.11s) kafka          : Kafka seems to be in order
SUCCESS (Took   0.00s) redis          : Redis is up and using 4.83M memory
SUCCESS (Took   0.02s) postgres       : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK Successfully got a user from postgres
SUCCESS (Took   0.01s) couch          : Successfully queried an arbitrary couch view
FAILURE (Took   0.00s) celery         : analytics_queue has been blocked for 18:49:59.521897 (max allowed is 0:30:00)
async_restore_queue has been blocked for 18:50:08.388927 (max allowed is 0:01:00)
background_queue has been blocked for 18:49:59.264292 (max allowed is 0:10:00)
case_import_queue has been blocked for 18:49:58.454671 (max allowed is 0:01:00)
case_rule_queue has been blocked for 18:50:08.378536 (max allowed is 1:00:00)
celery has been blocked for 18:50:08.366363 (max allowed is 0:01:00)
celery_periodic has been blocked for 18:49:59.451630 (max allowed is 0:10:00)
email_queue has been blocked for 18:49:58.556194 (max allowed is 0:00:30)
export_download_queue has been blocked for 18:50:00.925046 (max allowed is 0:00:30)
repeat_record_queue has been blocked for 18:49:59.579119 (max allowed is 1:00:00)
saved_exports_queue has been blocked for 18:49:59.464947 (max allowed is 6:00:00)
ucr_queue has been blocked for 18:50:08.353426 (max allowed is 1:00:00)
SUCCESS (Took   0.03s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.05s) blobdb         : Successfully saved a file to the blobdb
SUCCESS (Took   0.02s) formplayer     : Formplayer returned a 200 status code: https://inddex24.org/formplayer/serverup
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK
Connection to 10.3.0.4 closed.

No change after a
cchq monolith service celery restart

sudo supervisorctl status
reports everything is running:

commcare-hq-monolith-celery_flower                                                                                                                                                                                                RUNNING   pid 35442, uptime 0:23:51
commcare-hq-monolith-celery_repeat_record_queue,celery,case_import_queue,background_queue,export_download_queue,saved_exports_queue,analytics_queue,ucr_queue,async_restore_queue,email_queue,case_rule_queue,celery_periodic_0   RUNNING   pid 35589, uptime 0:23:41
commcare-hq-monolith-celerybeat                                                                                                                                                                                                   RUNNING   pid 51920, uptime 0:00:10
commcare-hq-monolith-django                                                                                                                                                                                                       RUNNING   pid 25165, uptime 0:37:08
commcare-hq-monolith-formsplayer-spring                                                                                                                                                                                           RUNNING   pid 33958, uptime 0:25:24
commcare-hq-monolith-pillowtop-AppDbChangeFeedPillow-0                                                                                                                                                                            RUNNING   pid 29954, uptime 0:28:36
commcare-hq-monolith-pillowtop-ApplicationToElasticsearchPillow-0                                                                                                                                                                 RUNNING   pid 29955, uptime 0:28:36
commcare-hq-monolith-pillowtop-CacheInvalidatePillow-0                                                                                                                                                                            RUNNING   pid 29956, uptime 0:28:36
commcare-hq-monolith-pillowtop-DefaultChangeFeedPillow-0                                                                                                                                                                          RUNNING   pid 29957, uptime 0:28:36
commcare-hq-monolith-pillowtop-DomainDbKafkaPillow-0                                                                                                                                                                              RUNNING   pid 29958, uptime 0:28:36
commcare-hq-monolith-pillowtop-KafkaDomainPillow-0                                                                                                                                                                                RUNNING   pid 29959, uptime 0:28:36
commcare-hq-monolith-pillowtop-LedgerToElasticsearchPillow-0                                                                                                                                                                      RUNNING   pid 29960, uptime 0:28:36
commcare-hq-monolith-pillowtop-SqlSMSPillow-0                                                                                                                                                                                     RUNNING   pid 29961, uptime 0:28:36
commcare-hq-monolith-pillowtop-UpdateUserSyncHistoryPillow-0                                                                                                                                                                      RUNNING   pid 29962, uptime 0:28:36
commcare-hq-monolith-pillowtop-UserCacheInvalidatePillow-0                                                                                                                                                                        RUNNING   pid 29963, uptime 0:28:36
commcare-hq-monolith-pillowtop-UserGroupsDbKafkaPillow-0                                                                                                                                                                          RUNNING   pid 29964, uptime 0:28:36
commcare-hq-monolith-pillowtop-case-pillow-0                                                                                                                                                                                      RUNNING   pid 29965, uptime 0:28:36
commcare-hq-monolith-pillowtop-group-pillow-0                                                                                                                                                                                     RUNNING   pid 29966, uptime 0:28:36
commcare-hq-monolith-pillowtop-user-pillow-0                                                                                                                                                                                      RUNNING   pid 29967, uptime 0:28:36
commcare-hq-monolith-pillowtop-xform-pillow-0                                                                                                                                                                                     RUNNING   pid 29968, uptime 0:28:36

If I open a python session and run:

python manage.py show_celery_tasks celery@10.3.0.4_repeat_record_queue,celery,case_import_queue,background_queue,export_download_queue,saved_exports_queue,analytics_queue,ucr_queue,async_restore_queue,email_queue,case_rule_queue,celery_periodic_0.1701687937_timestamp active
I get:
(none)

The celery log looks like this:

2023-12-04 11:05:51,012 INFO [celery.apps.worker] celery@10.3.0.4_repeat_record_queue,celery,case_import_queue,background_queue,export_download_queue,saved_exports_queue,analytics_queue,ucr_queue,async_restore_queue,email_queue,case_rule_queue,celery_periodic_0.1701687937_timestamp ready.
2023-12-04 11:10:03,390 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[982c4a75-1817-4ba5-9773-45b8d07b1aaf] received
2023-12-04 11:10:03,391 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:10:03,700 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[982c4a75-1817-4ba5-9773-45b8d07b1aaf] succeeded in 0.004233837999890966s: None
2023-12-04 11:10:51,022 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:15:03,192 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[864bed90-daa3-4236-a867-512310268bdd] received
2023-12-04 11:15:03,193 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:15:03,512 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[864bed90-daa3-4236-a867-512310268bdd] succeeded in 0.004273338999837506s: None
2023-12-04 11:15:51,033 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:20:03,141 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[be2e0719-869d-483d-b05a-adb9785e8153] received
2023-12-04 11:20:03,141 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:20:03,477 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[be2e0719-869d-483d-b05a-adb9785e8153] succeeded in 0.005137547000003906s: None
2023-12-04 11:20:51,046 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:23:02,150 INFO [celery.worker.strategy] Task corehq.apps.users.tasks.update_domain_date[fe2d1a47-5fb8-4653-9f80-6949b3ee3df8] received
2023-12-04 11:23:02,151 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:23:02,772 INFO [celery.app.trace] Task corehq.apps.users.tasks.update_domain_date[fe2d1a47-5fb8-4653-9f80-6949b3ee3df8] succeeded in 0.2613754040000913s: None
2023-12-04 11:23:51,051 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:25:03,113 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[9b1beb95-4899-4d58-b6a1-fca3643dcdc2] received
2023-12-04 11:25:03,113 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:25:03,485 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[9b1beb95-4899-4d58-b6a1-fca3643dcdc2] succeeded in 0.004864945000008447s: None
2023-12-04 11:25:51,055 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:30:03,539 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[938a32a4-035f-4d18-a670-a0abc8c927a0] received
2023-12-04 11:30:03,539 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:30:03,948 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[938a32a4-035f-4d18-a670-a0abc8c927a0] succeeded in 0.005046146000040608s: None
2023-12-04 11:30:51,068 INFO [celery.worker.autoscale] Scaling down 1 processes.
2023-12-04 11:35:03,112 INFO [celery.worker.strategy] Task corehq.apps.analytics.tasks._track_workflow_task[809026c2-af06-4460-b077-1751e0509efa] received
2023-12-04 11:35:03,112 INFO [celery.worker.autoscale] Scaling up 1 processes.
2023-12-04 11:35:03,469 INFO [celery.app.trace] Task corehq.apps.analytics.tasks._track_workflow_task[809026c2-af06-4460-b077-1751e0509efa] succeeded in 0.005102047000036691s: None
2023-12-04 11:35:51,081 INFO [celery.worker.autoscale] Scaling down 1 processes.

It's now 11:35 UTC so those are fairly recent activities and every now and then something else pops up on the log, but I'm not seeing the regular flurry of heartbeat entries I'm familiar with.

Any ideas very welcome!
Thanks

2

Hi @erobinson

Interesting.

  • Are you able to export cases/forms?
  • Does celery flower show anything unusual?
  • Did you confirm that Redis has enough memory left? I think we would have seen this manisfest a bit clearer if there were issues, but worth it to confirm I'd think.
1 Like
3

It seems I am able to export OK.

Redis INFO memory shows:

used_memory_human:8.21M
used_memory_rss_human:17.50M
used_memory_peak_human:12.82M
used_memory_peak_perc:64.01%
used_memory_overhead:1796586
used_memory_startup:858760
used_memory_dataset:6807742
used_memory_dataset_perc:87.89%
allocator_allocated:8963520
allocator_active:9863168
allocator_resident:12886016
total_system_memory_human:31.28G
used_memory_lua:32768
used_memory_vm_eval:32768
used_memory_lua_human:32.00K
used_memory_scripts_eval:496
number_of_cached_scripts:2
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total_human:64.00K
used_memory_functions:184
used_memory_scripts_human:680B
maxmemory_human:26.16G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.10
allocator_frag_bytes:899648
allocator_rss_ratio:1.31
allocator_rss_bytes:3022848
rss_overhead_ratio:1.42
rss_overhead_bytes:5459968
mem_fragmentation_ratio:2.14
mem_fragmentation_bytes:9762328
mem_not_counted_for_evict:1536
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:61258
mem_cluster_links:0
mem_aof_buffer:1536
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

I restarted celery and checked flower log:

2023-12-05 13:06:41,145 INFO [flower.command] 15 detected, shutting down
2023-12-05 13:07:24,185 INFO [flower.command] Visit me at http://0.0.0.0:5555
2023-12-05 13:07:24,192 INFO [flower.command] Broker: redis://localhost:6379/0
2023-12-05 13:07:24,257 INFO [flower.command] Registered tasks:
['casexml.apps.phone.tasks.get_async_restore_payload',
 'casexml.apps.phone.tasks.prune_synclogs',
 'celery.accumulate',
 'celery.backend_cleanup',
 'celery.chain',
 'celery.chord',
 'celery.chord_unlock',
 'celery.chunks',
 'celery.group',
 'celery.map',
 'celery.starmap',
 'corehq.apps.accounting.tasks.archive_logos',
 'corehq.apps.accounting.tasks.calculate_users_in_all_domains',
 'corehq.apps.accounting.tasks.calculate_web_users_in_all_billing_accounts',
 'corehq.apps.accounting.tasks.check_credit_line_balances',
 'corehq.apps.accounting.tasks.create_wire_credits_invoice',
 'corehq.apps.accounting.tasks.generate_invoices',
 'corehq.apps.accounting.tasks.pay_autopay_invoices',
 'corehq.apps.accounting.tasks.remind_dimagi_contact_subscription_ending_60_days',
 'corehq.apps.accounting.tasks.remind_subscription_ending',
 'corehq.apps.accounting.tasks.restore_logos',
 'corehq.apps.accounting.tasks.run_downgrade_process',
 'corehq.apps.accounting.tasks.send_autopay_failed',
 'corehq.apps.accounting.tasks.send_credits_on_hq_report',
 'corehq.apps.accounting.tasks.send_prepaid_credits_export',
 'corehq.apps.accounting.tasks.send_purchase_receipt',
 'corehq.apps.accounting.tasks.update_exchange_rates',
 'corehq.apps.accounting.tasks.update_subscriptions',
 'corehq.apps.accounting.tasks.weekly_digest',
 'corehq.apps.aggregate_ucrs.tasks.populate_aggregate_table_data_task',
 'corehq.apps.analytics.tasks._track_workflow_task',
 'corehq.apps.analytics.tasks.cleanup_blocked_hubspot_contacts',
 'corehq.apps.analytics.tasks.generate_partner_reports',
 'corehq.apps.analytics.tasks.identify',
 'corehq.apps.analytics.tasks.send_hubspot_form_task',
 'corehq.apps.analytics.tasks.track_built_app_on_hubspot',
 'corehq.apps.analytics.tasks.track_clicked_deploy_on_hubspot',
 'corehq.apps.analytics.tasks.track_clicked_signup_on_hubspot',
 'corehq.apps.analytics.tasks.track_confirmed_account_on_hubspot',
 'corehq.apps.analytics.tasks.track_job_candidate_on_hubspot',
 'corehq.apps.analytics.tasks.track_periodic_data',
 'corehq.apps.analytics.tasks.track_user_sign_in_on_hubspot',
 'corehq.apps.analytics.tasks.update_hubspot_properties',
 'corehq.apps.analytics.tasks.update_subscription_properties_by_domain',
 'corehq.apps.analytics.tasks.update_subscription_properties_by_user',
 'corehq.apps.api.tasks.clean_api_access',
 'corehq.apps.app_manager.tasks.autogenerate_build_task',
 'corehq.apps.app_manager.tasks.create_build_files_for_all_app_profiles',
 'corehq.apps.app_manager.tasks.create_usercases',
 'corehq.apps.app_manager.tasks.load_appcues_template_app',
 'corehq.apps.app_manager.tasks.prune_auto_generated_builds',
 'corehq.apps.app_manager.tasks.update_linked_app_and_notify_task',
 'corehq.apps.callcenter.tasks.calculate_indicators',
 'corehq.apps.callcenter.tasks.sync_usercases_task',
 'corehq.apps.case_importer.tasks.bulk_import_async',
 'corehq.apps.case_importer.tasks.store_failed_task_result',
 'corehq.apps.case_importer.tracking.analytics.get_case_upload_files_total_bytes',
 'corehq.apps.case_search.tasks.delete_case_search_cases_for_domain',
 'corehq.apps.case_search.tasks.reindex_case_search_for_domain',
 'corehq.apps.cleanup.tasks.check_for_conflicting_domains',
 'corehq.apps.cleanup.tasks.check_for_elasticsearch_data_without_existing_domain',
 'corehq.apps.cleanup.tasks.check_for_sql_cases_without_existing_domain',
 'corehq.apps.cleanup.tasks.check_for_sql_forms_without_existing_domain',
 'corehq.apps.cleanup.tasks.check_for_ucr_tables_without_existing_domain',
 'corehq.apps.cleanup.tasks.clear_expired_sessions',
 'corehq.apps.cleanup.tasks.permanently_delete_eligible_data',
 'corehq.apps.custom_data_fields.tasks.refresh_es_for_profile_users',
 'corehq.apps.data_analytics.tasks.build_last_month_GIR',
 'corehq.apps.data_analytics.tasks.build_last_month_MALT',
 'corehq.apps.data_analytics.tasks.update_current_MALT',
 'corehq.apps.data_analytics.tasks.update_malt',
 'corehq.apps.data_interfaces.tasks.bulk_case_copy_async',
 'corehq.apps.data_interfaces.tasks.bulk_case_reassign_async',
 'corehq.apps.data_interfaces.tasks.bulk_form_management_async',
 'corehq.apps.data_interfaces.tasks.bulk_upload_cases_to_group',
 'corehq.apps.data_interfaces.tasks.delete_duplicates_for_cases',
 'corehq.apps.data_interfaces.tasks.delete_old_rule_submission_logs',
 'corehq.apps.data_interfaces.tasks.reset_and_backfill_deduplicate_rule_task',
 'corehq.apps.data_interfaces.tasks.run_case_update_rules',
 'corehq.apps.data_interfaces.tasks.run_case_update_rules_for_domain',
 'corehq.apps.data_interfaces.tasks.run_case_update_rules_for_domain_and_db',
 'corehq.apps.data_interfaces.tasks.run_case_update_rules_on_save',
 'corehq.apps.data_interfaces.tasks.task_generate_ids_and_operate_on_payloads',
 'corehq.apps.data_interfaces.tasks.task_operate_on_payloads',
 'corehq.apps.domain.tasks.self_starter_email',
 'corehq.apps.dropbox.tasks.upload',
 'corehq.apps.enterprise.tasks.auto_deactivate_mobile_workers',
 'corehq.apps.enterprise.tasks.clear_enterprise_permissions_cache_for_all_users',
 'corehq.apps.enterprise.tasks.email_enterprise_report',
 'corehq.apps.events.tasks.close_mobile_worker_attendee_cases',
 'corehq.apps.events.tasks.sync_mobile_worker_attendees',
 'corehq.apps.export.tasks._start_export_task',
 'corehq.apps.export.tasks.add_inferred_export_properties',
 'corehq.apps.export.tasks.generate_schema_for_all_builds',
 'corehq.apps.export.tasks.populate_export_download_task',
 'corehq.apps.export.tasks.process_populate_export_tables',
 'corehq.apps.export.tasks.saved_exports',
 'corehq.apps.fixtures.tasks.async_fixture_download',
 'corehq.apps.fixtures.tasks.fixture_upload_async',
 'corehq.apps.hqadmin.tasks.check_pillows_for_rewind',
 'corehq.apps.hqadmin.tasks.count_es_cases_past_window',
 'corehq.apps.hqadmin.tasks.count_es_forms_past_window',
 'corehq.apps.hqadmin.tasks.create_historical_checkpoints',
 'corehq.apps.hqadmin.tasks.reconcile_es_cases',
 'corehq.apps.hqadmin.tasks.reconcile_es_forms',
 'corehq.apps.hqadmin.tasks.send_abnormal_usage_alert',
 'corehq.apps.hqadmin.tasks.send_mass_emails',
 'corehq.apps.hqadmin.tasks.track_es_doc_counts',
 'corehq.apps.hqadmin.tasks.track_pg_limits',
 'corehq.apps.hqcase.tasks.delete_exploded_case_task',
 'corehq.apps.hqcase.tasks.explode_case_task',
 'corehq.apps.hqmedia.tasks.build_application_zip',
 'corehq.apps.hqmedia.tasks.process_bulk_upload_zip',
 'corehq.apps.hqwebapp.tasks.alert_bounced_system_emails',
 'corehq.apps.hqwebapp.tasks.clean_expired_transient_emails',
 'corehq.apps.hqwebapp.tasks.clear_expired_oauth_tokens',
 'corehq.apps.hqwebapp.tasks.mail_admins_async',
 'corehq.apps.hqwebapp.tasks.process_bounced_emails',
 'corehq.apps.hqwebapp.tasks.send_html_email_async',
 'corehq.apps.hqwebapp.tasks.send_mail_async',
 'corehq.apps.linked_domain.tasks.pull_missing_multimedia_for_app_and_notify_task',
 'corehq.apps.linked_domain.tasks.push_models',
 'corehq.apps.linked_domain.tasks.release_domain',
 'corehq.apps.linked_domain.tasks.send_linked_domain_release_email',
 'corehq.apps.locations.tasks.delete_locations_related_rules',
 'corehq.apps.locations.tasks.download_locations_async',
 'corehq.apps.locations.tasks.import_locations_async',
 'corehq.apps.locations.tasks.sync_administrative_status',
 'corehq.apps.locations.tasks.update_users_at_locations',
 'corehq.apps.products.tasks.import_products_async',
 'corehq.apps.registration.tasks.activation_24hr_reminder_email',
 'corehq.apps.registration.tasks.delete_old_async_signup_requests',
 'corehq.apps.registration.tasks.send_domain_registration_email',
 'corehq.apps.reports.tasks.apps_update_calculated_properties',
 'corehq.apps.reports.tasks.build_form_multimedia_zip',
 'corehq.apps.reports.tasks.build_form_multimedia_zipfile',
 'corehq.apps.reports.tasks.delete_old_query_hash',
 'corehq.apps.reports.tasks.export_all_rows_task',
 'corehq.apps.reports.tasks.run_datadog_user_stats',
 'corehq.apps.reports.tasks.update_calculated_properties',
 'corehq.apps.reports.tasks.update_calculated_properties_for_domains',
 'corehq.apps.reports.util.sync_all_tableau_users',
 'corehq.apps.saved_reports.tasks.initiate_queue_scheduled_reports',
 'corehq.apps.saved_reports.tasks.purge_old_scheduled_report_logs',
 'corehq.apps.saved_reports.tasks.queue_scheduled_reports',
 'corehq.apps.saved_reports.tasks.send_email_report',
 'corehq.apps.saved_reports.tasks.send_report',
 'corehq.apps.saved_reports.tasks.send_report_throttled',
 'corehq.apps.settings.tasks.notify_about_to_expire_api_keys',
 'corehq.apps.sms.tasks.delete_phone_numbers_for_owners',
 'corehq.apps.sms.tasks.process_sms',
 'corehq.apps.sms.tasks.publish_sms_change',
 'corehq.apps.sms.tasks.queued_sms',
 'corehq.apps.sms.tasks.store_billable',
 'corehq.apps.sms.tasks.sync_user_phone_numbers',
 'corehq.apps.smsbillables.tasks.send_gateway_fee_report_out',
 'corehq.apps.smsforms.tasks.close_session',
 'corehq.apps.smsforms.tasks.handle_due_survey_action',
 'corehq.apps.smsforms.tasks.send_first_message',
 'corehq.apps.sso.tasks.create_rollover_service_provider_x509_certificates',
 'corehq.apps.sso.tasks.idp_cert_expires_reminder',
 'corehq.apps.sso.tasks.renew_service_provider_x509_certificates',
 'corehq.apps.toggle_ui.tasks.generate_toggle_csv_download',
 'corehq.apps.translations.tasks.backup_project_from_transifex',
 'corehq.apps.translations.tasks.delete_resources_on_transifex',
 'corehq.apps.translations.tasks.email_project_from_hq',
 'corehq.apps.translations.tasks.migrate_project_on_transifex',
 'corehq.apps.translations.tasks.pull_translation_files_from_transifex',
 'corehq.apps.translations.tasks.push_translation_files_to_transifex',
 'corehq.apps.user_importer.tasks.import_users_and_groups',
 'corehq.apps.user_importer.tasks.parallel_import_task',
 'corehq.apps.user_importer.tasks.parallel_user_import',
 'corehq.apps.userreports.tasks.async_indicators_metrics',
 'corehq.apps.userreports.tasks.build_async_indicators',
 'corehq.apps.userreports.tasks.build_indicators_with_agg_queue',
 'corehq.apps.userreports.tasks.delete_data_source_task',
 'corehq.apps.userreports.tasks.export_ucr_async',
 'corehq.apps.userreports.tasks.queue_async_indicators',
 'corehq.apps.userreports.tasks.rebuild_indicators',
 'corehq.apps.userreports.tasks.rebuild_indicators_in_place',
 'corehq.apps.userreports.tasks.resume_building_indicators',
 'corehq.apps.userreports.tasks.run_queue_async_indicators_task',
 'corehq.apps.users.tasks._rebuild_case_with_retries',
 'corehq.apps.users.tasks._remove_indices_from_deleted_cases_task',
 'corehq.apps.users.tasks.apply_correct_demo_mode_to_loadtest_user',
 'corehq.apps.users.tasks.bulk_download_usernames_async',
 'corehq.apps.users.tasks.bulk_download_users_async',
 'corehq.apps.users.tasks.process_reporting_metadata_staging',
 'corehq.apps.users.tasks.remove_unused_custom_fields_from_users_task',
 'corehq.apps.users.tasks.remove_users_test_cases',
 'corehq.apps.users.tasks.resend_pending_invitations',
 'corehq.apps.users.tasks.reset_demo_user_restore_task',
 'corehq.apps.users.tasks.tag_cases_as_deleted_and_remove_indices',
 'corehq.apps.users.tasks.tag_forms_as_deleted_rebuild_associated_cases',
 'corehq.apps.users.tasks.tag_system_forms_as_deleted',
 'corehq.apps.users.tasks.turn_on_demo_mode_task',
 'corehq.apps.users.tasks.undelete_system_forms',
 'corehq.apps.users.tasks.update_domain_date',
 'corehq.blobs.tasks.delete_expired_blobs',
 'corehq.celery_monitoring.heartbeat.heartbeat__analytics_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__async_restore_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__background_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__case_import_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__case_rule_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__celery',
 'corehq.celery_monitoring.heartbeat.heartbeat__celery_periodic',
 'corehq.celery_monitoring.heartbeat.heartbeat__email_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__export_download_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__repeat_record_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__saved_exports_queue',
 'corehq.celery_monitoring.heartbeat.heartbeat__ucr_queue',
 'corehq.form_processor.tasks._reprocess_archive_stubs',
 'corehq.form_processor.tasks.reprocess_archive_stubs',
 'corehq.form_processor.tasks.reprocess_submission',
 'corehq.messaging.scheduling.tasks.delete_alert_schedule_instances',
 'corehq.messaging.scheduling.tasks.delete_case_alert_schedule_instances',
 'corehq.messaging.scheduling.tasks.delete_case_timed_schedule_instances',
 'corehq.messaging.scheduling.tasks.delete_schedule_instances_for_cases',
 'corehq.messaging.scheduling.tasks.delete_timed_schedule_instances',
 'corehq.messaging.scheduling.tasks.handle_alert_schedule_instance',
 'corehq.messaging.scheduling.tasks.handle_case_alert_schedule_instance',
 'corehq.messaging.scheduling.tasks.handle_case_timed_schedule_instance',
 'corehq.messaging.scheduling.tasks.handle_timed_schedule_instance',
 'corehq.messaging.scheduling.tasks.refresh_alert_schedule_instances',
 'corehq.messaging.scheduling.tasks.refresh_timed_schedule_instances',
 'corehq.messaging.smsbackends.telerivet.tasks.process_incoming_message',
 'corehq.messaging.tasks.run_messaging_rule',
 'corehq.messaging.tasks.run_messaging_rule_for_shard',
 'corehq.messaging.tasks.set_rule_complete',
 'corehq.messaging.tasks.sync_case_chunk_for_messaging_rule',
 'corehq.messaging.tasks.sync_case_for_messaging_rule',
 'corehq.messaging.tasks.sync_case_for_messaging_task',
 'corehq.motech.dhis2.tasks.send_datasets',
 'corehq.motech.dhis2.tasks.send_datasets_for_all_domains',
 'corehq.motech.fhir.tasks.run_daily_importers',
 'corehq.motech.fhir.tasks.run_importer',
 'corehq.motech.fhir.tasks.run_monthly_importers',
 'corehq.motech.fhir.tasks.run_weekly_importers',
 'corehq.motech.openmrs.tasks.import_patients',
 'corehq.motech.openmrs.tasks.import_patients_with_importer',
 'corehq.motech.openmrs.tasks.poll_openmrs_atom_feeds',
 'corehq.motech.openmrs.tasks.track_changes',
 'corehq.motech.repeaters.dbaccessors.get_overdue_repeat_record_count',
 'corehq.motech.repeaters.tasks.check_repeaters',
 'corehq.motech.repeaters.tasks.check_repeaters_in_partition',
 'corehq.motech.repeaters.tasks.delete_old_request_logs',
 'corehq.motech.repeaters.tasks.process_repeat_record',
 'corehq.motech.repeaters.tasks.process_repeater',
 'corehq.motech.repeaters.tasks.retry_process_repeat_record',
 'corehq.pillows.tasks.fix_user_types',
 'corehq.pillows.tasks.resave_es_forms_with_unknown_user_type',
 'corehq.pillows.tasks.send_unknown_user_type_stats',
 'corehq.preindex.tasks.preindex_couch_views',
 'corehq.preindex.tasks.run_continuous_indexing_task',
 'custom.covid.tasks.clear_formplayer_db_for_user',
 'custom.covid.tasks.prime_formplayer_db_for_user',
 'custom.covid.tasks.prime_formplayer_dbs',
 'custom.onse.tasks._update_facility_cases_from_dhis2_data_elements',
 'custom.onse.tasks.update_facility_cases_from_dhis2_data_elements',
 'phonelog.tasks.purge_old_device_report_entries',
 'phonelog.tasks.send_device_log_to_sumologic',
 'pillow_retry.tasks.record_pillow_error_queue_size',
 'pillowtop.tasks.pillow_datadog_metrics']
2023-12-05 13:07:24,263 INFO [kombu.mixins] Connected to redis://localhost:6379/0
2023-12-05 13:07:25,338 WARNING [flower.inspector] Inspect method registered failed
2023-12-05 13:07:25,343 WARNING [flower.inspector] Inspect method stats failed
2023-12-05 13:07:25,344 WARNING [flower.inspector] Inspect method active failed
2023-12-05 13:07:25,345 WARNING [flower.inspector] Inspect method revoked failed
2023-12-05 13:07:25,350 WARNING [flower.inspector] Inspect method reserved failed
2023-12-05 13:07:25,350 WARNING [flower.inspector] Inspect method active_queues failed
2023-12-05 13:07:25,350 WARNING [flower.inspector] Inspect method scheduled failed
2023-12-05 13:07:25,351 WARNING [flower.inspector] Inspect method conf failed

I restarted it again, and this time none of the warnings at the end, it just ends with:
2023-12-05 13:07:24,263 INFO [kombu.mixins] Connected to redis://localhost:6379/0

Still no joy on the check_services though:

Ubuntu 22.04.3 LTS
SUCCESS (Took   0.11s) kafka          : Kafka seems to be in order
SUCCESS (Took   0.00s) redis          : Redis is up and using 9.78M memory
SUCCESS (Took   0.01s) postgres       : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK Successfully got a user from postgres
SUCCESS (Took   0.01s) couch          : Successfully queried an arbitrary couch view
FAILURE (Took   0.01s) celery         : analytics_queue has been blocked for 1 day, 20:40:41.163290 (max allowed is 0:30:00)
async_restore_queue has been blocked for 1 day, 20:40:50.030630 (max allowed is 0:01:00)
background_queue has been blocked for 1 day, 20:40:40.906023 (max allowed is 0:10:00)
case_import_queue has been blocked for 1 day, 20:40:40.096442 (max allowed is 0:01:00)
case_rule_queue has been blocked for 1 day, 20:40:50.020344 (max allowed is 1:00:00)
celery has been blocked for 1 day, 20:40:50.008217 (max allowed is 0:01:00)
celery_periodic has been blocked for 1 day, 20:40:41.093484 (max allowed is 0:10:00)
email_queue has been blocked for 1 day, 20:40:40.198087 (max allowed is 0:00:30)
export_download_queue has been blocked for 1 day, 20:40:42.566981 (max allowed is 0:00:30)
repeat_record_queue has been blocked for 1 day, 20:40:41.221101 (max allowed is 1:00:00)
saved_exports_queue has been blocked for 1 day, 20:40:41.106974 (max allowed is 6:00:00)
ucr_queue has been blocked for 1 day, 20:40:49.995574 (max allowed is 1:00:00)
SUCCESS (Took   0.04s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.01s) blobdb         : Successfully saved a file to the blobdb
SUCCESS (Took   0.01s) formplayer     : Formplayer returned a 200 status code: https://inddex24.org/formplayer/serverup
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK
4

Just to add, the only thing that's changed is the VM that the machine runs on was scaled down from an Azure instance with 56Gb RAM to one with 32Gb RAM. I do wonder if there is some calculation that goes into the deployment process where RAM limits are assigned to the various services. Is it possible that the downscale has had an effect on the system?

5

This issue was resolved when deploying an updated build. Thanks for the input!