Post xls file to commcare from buffer

Developers
#1

Hi all,

I'm having some issues sending data to CommCareHQ. The data I am trying to upload is an xls file with the endpoint bulk_upload_api (Bulk Upload Case Data - CommCare Public - Dimagi Confluence) using axios. For some reason, I don't have to store the file in the filesystem, but I'm sending the streams (build from a buffer using the npm package xlsx).
The response I'm having from CommCare is this line:

 data: {
      code: 500,
      message: "Error: Invalid POST request. Both 'file' and 'case_type' are required"
    }

...even though I append the file (buffer) and the case_type. Check snippet below:

let data = new FormData();
data.append('file', buffer);
data.append('case_type', 'xxx');
data.append('search_field', 'external_id');
data.append('create_new_cases', 'on');

Anyone has evey tried this way of uploading case and could help?

#2

Some additional context that's important here:

  1. We're using NodeJs (via OpenFn) and have tested request, axios, and the underlying http module using form-data. None of them seem to work, though we can replicate the exact output of a successful file upload (see point 2) when testing on a mock server.
  2. We're able to upload using curl and a locally stored file, but when doing so via NodeJs and axios we a very strange 405. When posting to https://www.commcarehq.org/a/%5Bdomain%5D/importer/excel/bulk_upload_api/ we get a 405 and it's clear that there's some sort of redirect going on that results in a GET being executed. In order to get the 500 we need to append the port to the above url: https://www.commcarehq.org:443/a/%5Bdomain%5D/importer/excel/bulk_upload_api/. That doesn't seem quite right, but multipart/form-data is certainly not my speciality! Is there something we're missing?
  3. See images below for what we're sending (when testing on RequestBin). Does this seem like what's being specified in the API?
    image
    image1042×220 26.4 KB

    image
    image1003×389 27.2 KB

Thanks in advance for your help!

#3

Here's the code that triggering the error. My guess is that whatever the JavaScript implementation is doing is causing the data to all land in django's request.FILES instead of request.POST or vice-versa, but not be properly split across them. You might want to explore playing with enctype versus content-type headers. More docs on how Django handles this are here and here.

It might be that CommCare should also be more flexible in allowing variants of these requests to be supported. If you are able to find a fix, please report back so we can improve the docs in the meantime!

1 Like
#4

It's all solved Cory. Thanks for digging into this.
For what it's worth, we should add a filename {filename: 'file.xls=' to the buffer so it could be send as a file.
data.append(f'ile', buffer, {filename: 'file.xls'});