Use Data Export Tool to transfer data into postgres database (running in docker)

Developers
1

Hi everyone, I'm trying to use the Data Export Tool to schedule data transfers from the commcare plattform to a postgres database. Some additional background information: In our setup, the DET and the "targetdatabase" (in which the data should be pushed) run in separate docker containers on my local machine (eventually on AWS). I was able to generate an API key through the project space in commcarehq, and in the DET itself, I set up our project, specified the database and one export... This export fails with "psycopg2.OperationalError: could not connect to server: Connection refused", meaning the targetdatabase can't be reached. From the top of my head, I can see two potential causes: Wrong IP address when creating the API key, or wrong specs in the database creation via DET. However, even if it's one of these cases, I currently don't exactly know how to fix either one.

My questions here are
a) Has anyone faced a similar issue before in this setup (push to postgres db in other docker container) and has some tips how to solve it?
b) Does anyone have some general tips on debugging the system in cases like this (Reverse engineering everything from the ground up seems quite impractical in this situation)?

2

Hi Jan

"psycopg2.OperationalError: could not connect to server: Connection refused",

This error indicates the the DET is unable to connect to your target database. This is likely caused by your docker setup. Here are some followup questions:

  1. Are you using Docker Compose to run the containers or are you running them independently?
  2. Have you exposed the correct ports for the postgres container (5432 by default for PostgreSQL)

If you are not using Docker Compose you may need to setup a bridge network. This article may also be useful: How To Communicate Between Docker Containers - Tutorial Works

3

Hi Simon, thanks so much for your answer! We had run the containers independently and I looked into connecting them via a network bridge, but ended up simply putting DET and my DB container in the same docker compose file, which seemed to work!