we(echis) have been developing many reports using commcarehq for the past 2+ years. but after some time the ucr keeps on returning incorrect result, checked it by exporting cases using data export feature. for instance the ucr is not picking up cases that were closed using the import feature. After manually ordering a rebuild on the ucr the count is right. Do we have to order the rebuild manually periodically?
Hi Andinet,
In general, it is not expected that you'd have to manually rebuild UCRs. In practice, this can happen and so doing so - if feasible - is not a bad thing to do. It's especially recommended after changes are made to the UCR unless you don't need the changes to appear on old data.
The real-time updates to UCRs are based off the CommCare change feeds, and handled by a "Pillow". Most reasons data would be missing would be either a UCR configuration error, or an infrastructure issue somewhere in the web --> change feed --> pillow pipeline.
Common reasons they might not contain all data:
- There were errors with the UCR causing the saving to fail that were later fixed, but the document was not reprocessed.
- There was an error writing the change to Kafka when the document was created. E.g. if a Case import ran successfully, but Kafka was down at the time.
- Kafka lost data before the Pillows were able to process it (either due to a storage failure or due to the pillow being down longer than the Kafka retention period)
Hope that helps! Without knowing any more specifics or doing a deeper dive it's hard to know specifically why they are missing data in any specific instance.