Running commcare data export --since or automatic incremental


(Jos van der Ent) #1

Hi All,
I am currently migrating our database from Postgres to Sql server on Azure. I have the export tool scripted to run every hour with --since parameter and assumed the incremental filling was used (as stated in the documentation. Our base data is in the sql server, also data corrections (e.g. typing errors etc) are done in the database, so we need to be sure the corrections are not undone/overwritten by the export tool.
During the migration I became a bit confused on the auto increment option and --since parameter when running the data export tool.

  • From what I understood you can provide a since statement and any records from before that point won’t be transferred (commcare_export_runs table).
  • Than there is also the auto increment, that the tool only transfers newer entries since last (successful) run.

Now when I ran the new updated version of export tool a warning message was shown:

2019-01-07 15:46:14,416 commcare_export.cli WARNING Checkpointing disabled when using ‘–since’ or ‘–until’

Q1 Are the above statements correct?

Q2 also assuming one should choose between --since parameter or auto increment: if one chooses auto increment and changes are made to the excel reference file, the system will find a new md5 hash and so reload all data. Is that correct and could it be avoided?

How do other persons here run the export tool?
e.g. Running the script hourly and make the since statement move along with the current date (e.g. yesterday) and considering the second question how to avoid the tool to rerun everything once the md5 hash is changed e.g. due to updated Excel file?

– update: was able to reproduce the error message, so added it to question above