Commcare-export tool: unable to filter based on date

Developers
1

Hi,

I recently tried to export case data using commcare-export tool.To filter only 2021 data, i used --since 2021-01-01 --until 2021-04-01 parameters

But it populates the whole data as shown below: ignoring the date filter

2021-04-05 14:29:14,611 commcare_export.commcare_hq_client DEBUG Fetching 'case' batch: {'type': 'client', 'limit': 200, 'indexed_on_start': '2021-01-11T23:00:53.823114', 'indexed_on_end': '2021-04-01T00:00:00', 'order_by': 'indexed_on'}
2021-04-05 14:29:15,585 commcare_export.commcare_hq_client DEBUG Received 11800 of 3038130

2

Hi there,

This is related to a recent change to filtering that was made in version 1.5.0 to prevent data loss in certain scenarios. More details can be found in the linked release notes. However, if the date your data was indexed is far off from the date it came in, then you will get substantially different data. This could happen if you reindexed ElasticSearch.

A few short term workarounds:

  1. You could downgrade the DET to a version before 1.5
  2. You could manually set the pagination_mode column to "date_modified" in the commcare_export_runs table

In the future we should support --since and --until parameters supporting either pagination mode. I've created a Github issue here to track that: Support --since and --until using date_modified pagination · Issue #181 · dimagi/commcare-export · GitHub

3

Hi Cory,

Thanks for the prompt response.

I rolled back commcare-export to version 1.4.0 and able to fiter by date using --since and --until parameters. and got around 300k rows (it seems filtering works compared to 3milion data)

but we recently reindexed elastic search. so shall we check the data again?