Removing Datadog

Developers
1

I've updated my public.yml with DATADOG_ENABLED: false
I've also run an update-config and have deployed a new build since.
I noticed datadog mentioned in my syslog and realised I hadn't uninstalled the Datadog agent so I did that, but since uninstalling it, I have been getting a bunch of messages in the celery logs which appeared right as I uninstalled the datadog agent:

[2022-03-08 08:31:00,609: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,610: INFO/ForkPoolWorker-11] Task corehq.celery_monitoring.heartbeat.heartbeat__saved_exports_queue[f0bcd0c9-c587-4ed1-9481-5da8c99$
[2022-03-08 08:31:00,611: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,613: INFO/MainProcess] Received task: corehq.celery_monitoring.heartbeat.heartbeat__email_queue[a90b7143-0306-4da6-b6d4-d36efcc780c$
[2022-03-08 08:31:00,615: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,616: INFO/ForkPoolWorker-11] Task corehq.celery_monitoring.heartbeat.heartbeat__email_queue[a90b7143-0306-4da6-b6d4-d36efcc780ca] s$
[2022-03-08 08:31:00,617: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,617: INFO/MainProcess] Received task: corehq.celery_monitoring.heartbeat.heartbeat__celery_periodic[d96a1979-4143-4634-8dba-21ba55b$
[2022-03-08 08:31:00,620: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,621: INFO/ForkPoolWorker-11] Task corehq.celery_monitoring.heartbeat.heartbeat__celery_periodic[d96a1979-4143-4634-8dba-21ba55b9ce4$
[2022-03-08 08:31:00,621: INFO/MainProcess] Received task: corehq.celery_monitoring.heartbeat.heartbeat__case_rule_queue[bc812eab-0401-4118-97f8-cfb3518$
[2022-03-08 08:31:00,622: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,625: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket
[2022-03-08 08:31:00,625: INFO/ForkPoolWorker-11] Task corehq.celery_monitoring.heartbeat.heartbeat__case_rule_queue[bc812eab-0401-4118-97f8-cfb351846aa$
[2022-03-08 08:31:00,626: WARNING/ForkPoolWorker-11] Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket

I assume these relate to Datadog? Is there another step I've missed?
I am able to sync with the server etc. otherwise.
Thanks!

EDIT I also ran a:
sudo apt-get remove --purge datadog-agent -y
and restarted celery... and the messages appear to have stopped. I'm not sure if that's coincidence or not.

I just want to confirm that these actions are OK - i.e. the Datadog agent is not required or installed by Ansible during a deploy with DATADOG_ENABLED=false in public.yml?

Thanks again!

EDIT I still see mention of datadog in my celery logs:

[2022-03-08 09:17:10,813: INFO/MainProcess] Received task: corehq.apps.reports.tasks.run_datadog_user_stats[d454a049-6429-4e5b-a904-5ddd4d8eda8f]  
[2022-03-08 09:18:00,002: INFO/MainProcess] Received task: pillowtop.tasks.pillow_datadog_metrics[51c8aecb-0090-4599-96b2-779d3973366d]  
[2022-03-08 09:18:00,002: INFO/MainProcess] Scaling up 1 processes.
[2022-03-08 09:18:01,537: INFO/ForkPoolWorker-14] Task pillowtop.tasks.pillow_datadog_metrics[51c8aecb-0090-4599-96b2-779d3973366d] succeeded in 1.2584319850429893s: None

..etc.

2

Hi Ed

Did you restart your services after you ran update-config? A restart is required to pick up the changes.

To verify that Datadog is disabled you can check that the METRICS_PROVIDERS Django setting is empty:

grep METRICS_PROVIDERS /home/cchq/<env>/current/localsettings.py

In terms of the celery tasks, that is expected. Although these are named 'datadog' they are generic metrics tasks and will send metrics to whatever provider is configured. If no provider is configured then it will fall back to logging the values to the commcare.metrics logger.

1 Like