Bulk upload case data via the API to create new cases

Users
1

So the overall plan here is that mobile workers would not be creating the cases, but rather we would be creating the cases from a .net webapp via the API. These cases would be owned by a location using an organizational structure and setting the owner_id of the case so that mobile workers assigned to a location will be able to access the cases. The mobile workers would then need to be able select a case from a case list within the commcare app and make updates to it via a followup form.

The organizational structure would look something similar to this:
Country
Region
District
FDP (Food Distribution Point)

I see that cases can be created via the API using the bulk upload case data. see link below:
https://confluence.dimagi.com/display/commcarepublic/Bulk+Upload+Case+Data

I have 3 Primary questions.

  1. Using this method of upload via the API would the Mobile workers be able to sync their commcare mobile apps and then update those cases via a followup form?

  2. What is the case upload limit with this method of case upload? For example a FDP may have up to 37 000 cases or as few as 1 case. Keeping in mind that there are multiple FDPs.

  3. What is the case limit that the mobile app can manage? Cases will be fairly simple, only recording things like:
    house id,
    head of household name and some uniquely identifying attributes,
    unique participant GUID

Would the commcare mobile app be able to manage 37 000 cases or should that be broken down further to lower locations?

I see there is also another method of uploading case data linked to individual x form submissions. Using the submission API. It would certainly be easier not to have to upload form data as well as case data by making use of the bulk upload case data. However I am not sure what the best method is here based on the info I have given above. Link to Submission API below:

https://confluence.dimagi.com/display/commcarepublic/Submission+API#SubmissionAPI-Casemanagement

Any information would be greatly appreciated.

@Ethan_Soergel
@Simon_Kelly
@Mazz
@Sheel_Shah
@erobinson

2

Hey Calvin

Sounds like you're doing some of the stuff that I am looking at myself. Would be great to have a chat about this sometime.

Here are my two cents:

TLDR: There shouldn't be a limit to how many cases you can have in your database but performance will vary. only way to be sure is test on intended hardware.

1- Syncing should work if the owner_id is something that the mobile worker should have access to. for that reason, any kind of updates via API's should be pretty solid. a case without an owner_id will be hard to find in your apps.

2- I do not believe there is a limit on the number of cases the database can host.
3- This is a 3 part answer: A- it's limited by the storage space you have I believe or some stupidly high number. but more importantly B- Your hardware will determine the practical limit. and most importantly C: How many case properties do you have and how complicated are your forms regarding references to these cases. a combination of these three factors will determine what is practical to do. I

I have had applications become unusable with as little as a few thousand fat cases. but, not until I opened a certain form that had a lot of calls to the casedb object in its calculations.

if you're running webapps on a browser I doubt that 37K cases will make it too slow. on a Samsung S23 Ultra, probably also not a problem. there is only one way to find out :smile:

There are a few techniques you can break them down like you mentioned by using the org structure. I have also observed that form behavior changes between question lists and groups. Display conditions also affect when calculations fire. using the Depends() function is helpful too. In the end you can also resort to having a "Are you done with data entry?" checkbox that will allow calculations to fire once the form is completed. this basically makes all the lag happen towards the end rather than frequently while the enumerator is collecting data.

HTH

Mazz

3

Hi there @Mazz,

Thank you so much for your response and my apologies for the late response. I havent been able to get back to this until now. It would be great to have a chat about it sometime. I really appreciate all the info you shared here. I still need to get to a proof of concept though and see that my plan is going to work. Will share here again when I do either way.

I guess I just wanted to make sure that before I go down this path that the bulk upload case data API would work for me in registering the initial cases for the mobile workers to then be able to complete a followup form on that case at a later stage in the process. I guess that is something that I am just going to have to test out to see if it works or not.

Kind Regards,

4

Hi @Calvin

I'm going to try to fill in some gaps that @Mazz might have left out:

Using the Bulk Upload API, would mobile workers be able to sync CommCare apps and then update those cases?

Yes, they would. If the mobile workers can see the location's cases, and the new cases are owned by the location, it should work exactly as you expect.

What is the case upload limit of the Bulk Upload API?

The limit is the number of rows that an "xlsx" Excel file allows, which I believe is 1,048,576.

What is the case limit that the mobile app can manage?

As @Mazz pointed out, that will depend on the mobile device, and the CommCare app. Keep in mind that you may encounter usability issues before you reach device limits.

Regarding the choice between the Bulk Upload API and the Submission API:

Under the hood, the Bulk Upload API is creating an XForm, and submitting it.

If you can create a XForm programmatically, then you may find the Submission API is faster for processing cases, and easier for you to send to, because you won't have to build an Excel file first. I would chunk the cases to maybe 100 per XForm. Building an XForm programmatically may seem arcane (it is) and daunting (it needn't be). You can skip the form questions and answers, and just include the <case> and <meta> tags.

You can find a code example (in Python) that reads case data from a CSV file, and builds and submits an XForm from it, here: GitHub - dimagi/submission_api_example: An example showing how to use the CommCare HQ Submission API

(Example script here: submission_api_example/submit_data.py at master · dimagi/submission_api_example · GitHub but read the repo's README first. (I haven't tried, but I bet ChatGPT could do a reasonable job of translating it into your language of choice.))

1 Like
5

Thank you so much for your response @Norman_Hooper. That answers all the remaining questions I had about this scenario. Really good to know for definite that the plan forward can and should work! I am looking forward to getting this project to work.