Error Rebuilding UCR

andyasne · May 3, 2024, 7:29pm

I am getting the following error while trying to rebuild a UCR. My UCR contains 562 Columns(indicators).

Is there a limit that is set on the UCR indicators?
Is the error related to the number of indicators or the records that I have. This UCR was rebuilding fine before I added more indicators and the data is the same.

mkangia · May 31, 2024, 4:10pm

Hi,

Can you share the URL where you are seeing this?

A feature was recently built in HQ, behind a feature flag so that it's only applicable for required projects.

github.com/dimagi/commcare-hq

Restrict data source rebuild on UI

dimagi:master ← dimagi:mk/3538-restrict-rebuild

opened 08:19PM - 06 Apr 24 UTC

mkangia

+189 -46

## Product Description For projects added to the feature flag, restrict them …from rebuilding data sources, if the number of records to be iterated for the data source is higher than a threshold. Keeping this threshold as 1M for now. This is to restrict projects with high data volume, from misusing rebuild on data source unknowingly. ## Technical Summary  https://dimagi.atlassian.net/browse/SC-3538 Added an extra commit to tone down the limit for testing since adding 1M records on a test space isn't great use of time. Will drop this commit before merging. ## Feature Flag `RESTRICT_DATA_SOURCE_REBUILD` & `USER_CONFIGURABLE_REPORTS` ## Safety Assurance ### Safety story  Tested locally that the check works as expected. Also tested on staging. Used ES instead of SQL (rebuild process using SQL) to get the number of records so that the count is fetched efficiently. Additionally the change is behind a feature flag. ### Automated test coverage ### QA Plan  Will go through QA. https://dimagi.atlassian.net/browse/QA-6337 ### Rollback instructions  - [X] This PR can be reverted after deploy with no further considerations ### Labels & Review - [X] Risk label is set correctly - [X] The set of people pinged as reviewers is appropriate for the level of risk of the change

This change limits projects from re-building large UCR data source, and that limit is currently set to 1 million records.

Rebuilding large data sources can put considerable load on a shared setup with multiple projects but if you are seeing this on an independent CommCare HQ setup that is run by you, you can disable the feature flag for your project space.
This can be done by going to Admin > Feature Flags from the navigation tab and then search for feature flag "restrict data source rebuilt from UI". You can then edit it to remove your project space.