Get form data modified since timestamp

Hi there,

I have a question from a batch data sync/integration perspective, where the goal is to retrieve form submissions which were created or modified since the timestamp of the last sync and the current time.

In that regard:

  1. Are CommCare form submissions immutable, i.e. can a form submission ever change once it is received by the server?

  2. A form submission in the API has a received_on field and a server_modified_on field. These field values seem to be almost equal but not quite. Can there be a situation where a form submission is first received by the server and then modified later?

  3. The List forms API has two query parameters received_on_start and received_on_end. Would querying the forms endpoint with a start and end timestamp give me the form submissions which were created or modified between those two timestamps, or is there a better way to achieve this?

Hey,

Thanks for the thoughtful questions and apologies that this is a confusing topic! I've just updated the documentation of the List Forms API to add two new parameters (indexed_on_start/end) and these are the recommended fields you should use for data synchronization. Also, if you are trying to export forms into a DB you should take a look at the Data Export Tool which handles much of the complexity for you.

To answer your specific questions:

  1. While it is unusual, form submissions can change e.g. from the "data cleaning" feature.
  2. Yes, see above.
  3. Yes, use indexed_on as per above.

Hope that helps,
Cory

Perfect. This explains it and is exactly what I need. Appreciate the quick reply.

Lars,
Hi, we met a few years ago at the annual DHIS2 Symposium.

Are you aware of CommCare Sync? Cory has led development on this tool which is a web application that has all the functionality of the Data Export tool that he mentioned plus more automated logging, scheduling, and multi-project exports that make this job easier when you have many CommCare domains that you want to store in one environment for analysis.

https://confluence.dimagi.com/display/commcarepublic/CommCare+Sync%3A+web+application+for+managing+your+data+warehouse

We also have some abilities to automate the generation of the query file that the Data Export Tool requires. And improved documentation on this process is coming out soon, very soon.