Commcare-export blending --since and incremental refresh

Users
1

Hi all, I have a total of 9 million rows in my CommCare case. Out of the total 9 millions only the recently collected 3 millions records are needed for analysis and data collection is ongoing.

As I need only the data after '2024-12-01' onwards, I plan to use the --since parameter and the whole command seems as below

commcare-export --output-format sql --output mssql+pyodbc://localhost/MyDB?driver=ODBC+Driver+17+for+SQL+Server --project myCommcareWorkspace --query myExcel.xlsx --missing-value NULL --batch-size 1000 --since 2024-12-01 --username myUserName --password myPassword

I tried it and it pulls all 3 millions data successfully. However, as data collection is on progress whenever I re-run the above script it doesn't bring only the newly added one rather everything from '2024-12-01'.

Is there a way to use the --since parameter for the first time to pull all recent data after 2024-12-01 up until now and then for the subsequent data pulling continue with incremental refresh?

2

is there any way you could make the date dynamic? like today() - 1?

I think you want to post this on the dev section too - this section is more for configuration of apps and forms and what not

if you COULD do something like that to the date bit, then just match that dynamic date with your polling frequency "once a week, once a day, or whatever it is you're polling at"

3

Thank you for your response.

Does the --since parameter accepts non-literal values like today()-1 ?
I tried it and it gives me an error .... dateutil.parser._parser.ParserError: Unknown string format: today()-1