[Resolved] Errors in deploy after enabling Datadog

Developers
1

I enabled datadog in public.yml, added my API and APP keys to the vault and performed a deploy. The check goes through fine but the deploy fails at the Datadog agent step:

PLAY [Datadog agent] ************************************************************************************************************************************************

TASK [add datadog integration configs] ******************************************************************************************************************************
skipping: [10.1.0.4] => (item={'name': 'airflow', 'enabled': False}) 
failed: [10.1.0.4] (item={'name': 'elastic', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c537cc7a168c6ffce3cc7185b6f0c5a2c79fb89", "item": {"enabled": true, "name": "elastic"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'gunicorn', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "5af0d5a62dfa6295d6f4b0cc5211291f6238a0b5", "item": {"enabled": true, "name": "gunicorn"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'kafka', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ef03b3f3e35adf1103a48b9c218bde09e14bf9bc", "item": {"enabled": true, "name": "kafka"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'kafka_consumer', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d8f576c13dc1dd959429fafd49fdafe4e9b139c0", "item": {"enabled": true, "name": "kafka_consumer"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'nginx', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "8fbd747ee6282e0dd8d2f3f3e43388218f28a55d", "item": {"enabled": true, "name": "nginx"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'pgbouncer', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "c307e480a5382cff612352010572560eee8a40a1", "item": {"enabled": true, "name": "pgbouncer"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
skipping: [10.1.0.4] => (item={'name': 'pgbouncer_custom', 'enabled': False}) 
failed: [10.1.0.4] (item={'name': 'postgres', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb9a470aba1bcbe40706b84470ee0449bd8cdbfc", "item": {"enabled": true, "name": "postgres"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
skipping: [10.1.0.4] => (item={'name': 'process', 'enabled': False}) 
skipping: [10.1.0.4] => (item={'name': 'rabbitmq', 'enabled': False}) 
failed: [10.1.0.4] (item={'name': 'redisdb', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "17a30f63fb976f688089123fb4e2c9cb791b3528", "item": {"enabled": true, "name": "redisdb"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'zk', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "50d504c9a097e480103df42b2b01b141113d1762", "item": {"enabled": true, "name": "zk"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'jmx', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "cf40bffc5c018897303548f101b830e0d276f27e", "item": {"enabled": true, "name": "jmx"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'celery', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "6b1818ce3f1f24202fe01fede2c8a47ab957cd77", "item": {"enabled": true, "name": "celery"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'couch', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "b368daeba00f9b1d70bf72bae83acc30de7940f6", "item": {"enabled": true, "name": "couch"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'couch_custom', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "937c88366a9bfa4b24e93455ec7454d343ea4cd3", "item": {"enabled": true, "name": "couch_custom"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
skipping: [10.1.0.4] => (item={'name': 'shell', 'enabled': False}) 
failed: [10.1.0.4] (item={'name': 'http_check', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ef8912abda0d006b6982fc01932890bdb97ad15c", "item": {"enabled": true, "name": "http_check"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
failed: [10.1.0.4] (item={'name': 'haproxy', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "f419e7b3622a4e784ebe117d5d262111078f6c00", "item": {"enabled": true, "name": "haproxy"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
skipping: [10.1.0.4] => (item={'name': 'tcp_check', 'enabled': False}) 
failed: [10.1.0.4] (item={'name': 'disk', 'enabled': True}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1c00d93a5eaa94badc6d6f67a470fadf905f65f4", "item": {"enabled": true, "name": "disk"}, "msg": "Destination directory /etc/dd-agent/conf.d does not exist"}
skipping: [10.1.0.4] => (item={'name': 'swift', 'enabled': []}) 

PLAY RECAP **********************************************************************************************************************************************************
10.1.0.4                   : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

✗ Apply failed with status code 2
>> Sending email: Formplayer deploy to monolith failed
Ubuntu 18.04.5 LTS
Connection to 10.1.0.4 closed.

I was contemplating creating those directories but assumed they would usually be created during the datadog agent install? Either way it can wait so I thought I'd post here first.
Thanks!

2

Hi Ed

That directory should be created when the Datadog agent is installed on the machine. You could check if the agent is actually installed.

What was the deploy command that you ran?

3

It was a full 'cchq monolith deploy' that I ran. The agent doesn't appear to be installed yet, I was running the deploy to install it (assuming it's installed by the Ansible system?)

4

Ah, I see what's happening. The code Formplayer code deploy has some Datadog tasks tagged to run as well.

You'll need to do the Datadog setup before you deploy:

cchq monolith ap deploy_common.yml --tags=datadog

That should install the agent and set everything up.

1 Like
5

Thanks Simon. In an install from scratch, with a reglar deploy (datadog enabled), would the datadog agent setup be taken care of or is this always required to be run like this?

6

Yes, a normal environment setup (cchq <env> deploy-stack) will install and configure datadog. Code deploys do not usually apply any other infrastructure changes outside of updating the CommCare / Formplayer services.

1 Like
8

I'm having trouble with the install:

TASK [Install datadog agent] ****************************************************************************************************************************************
fatal: [10.1.0.4]: FAILED! => {"changed": false, "msg": "No package matching 'datadog-agent' is available"}

Does Commcare use version 7 of the agent? Can I add the repo as per the instructions here: Datadog: Log In
?
Thanks!

EDIT what's odd is I see there is a step for adding the apt key and repo, so perhaps it's just the check failing because they weren't yet there... I assume I can just run the command with --skip-check in future.

It looks like we're in business again. Thanks again for the help!
Just to add - I love the dashboards...