[Action Required] Possible stale data in elasticsearch

A bug was recently introduced which resulted in a small percentage of forms and case not being properly replicated to elasticsearch on some environments. This means those documents would not appear in reports or exports. This issue has since been resolved, but if it affected your environment, you will need to retroactively trigger sending of those documents to elasticsearch.

These problems would have arisen after deploying CommCareHQ after March 12th, and should have been resolved on your first deploy after March 23rd. If you didn’t deploy in that window, this issue shouldn’t affect your environment.

To start, ensure you’re on the latest version of commcare-cloud

update-code

You should also deploy CommCareHQ if you haven’t done so recently.

cchq <env> deploy

Then use the stale_data_in_es command to find any documents missing or out-of-date in elasticsearch. Be sure to replace <env> with your environment name (eg, monolith). If your project has a very large number of form submissions, you can reduce the time the command takes to run by narrowing the date range to the window during which that code was live, though I’d add a day on either end to be sure.

cchq <env> django-manage stale_data_in_es form case --start 2020-03-12 --end 2020-04-03 --tmux --tee changes.csv

This will make a file called changes.csv in your HQ code directory containing a log of affected forms and cases. The command will leave you in tmux in your CommCareHQ code root. You may inspect the log of changes at this time by viewing changes.csv

Lastly, exit tmux (ctrl+d), then issue the command to republish the documents detailed in that log file:

cchq <env> django-manage republish_doc_changes changes.csv

Everything should now be up-to-date. You may follow this procedure at any time if you suspect forms and cases may be missing from elasticsearch.