Migration to S3 backend fails check_services on self-hosted S3 cluster

Developers
1

Hi team,

I am currently migrating our commcare-cloud deployment (Ubuntu 22.04) from a local File System blobdb backend to an S3-compatible backend using a self-hosted Garage S3 cluster behind an Nginx proxy.

i followed this documentation for the process. Migrate from File System backend to an S3 compatible backend — CommCareHQ Deployment documentation

Prior to running the automated scripts, I verified that the remote object storage cluster is fully reachable and functional by successfully executing commands with the AWS CLI, such as:

aws --endpoint-url=http://192.168.200.41 s3 cp test.txt s3://test/

image
image1920×515 181 KB

I added the following settings to my public.yml file:

s3_blob_db_enabled: yes

s3_blob_db_url: "http://192.168.200.41"

s3_blob_db_s3_bucket: misau
localsettings:

BLOB_DB_MIGRATING_FROM_FS_TO_S3: True

I also added the respective credentials to environments/<env>/vault.yml:

  • secrets.S3_ACCESS_KEY

  • secrets.S3_SECRET_KEY

After configuring these, I ran commcare-cloud <env> update-config and restarted the commcare services. However, running commcare-cloud <env> check_services throws the following exception under blobdb

EXCEPTION (Took 0.08s) blobdb : Service check errored with exception 'ClientError('An error occurred (400) when calling the HeadBucket operation: Bad Request')'

Because Garage strictly requires path-style addressing (rather than virtual-host style routing), it seems the underlying boto3 client initialized by CommCare is defaulting to virtual-host lookups and causing the 400 error.

Is there an additional configuration block or environment setting needed in public.yml to ensure boto3 maps the HeadBucket check properly for a custom path-style S3 endpoint, or is this a known limitation of the automated check_services script with non-AWS S3 endpoints?

Thanks in advance!

2

Hi @demis08 ,

I don't think this is path-style addressing. HQ already sends path-style requests to custom endpoints (HEAD http:///misau).

It might be region mismatch.

HQ doesn't set a region on its S3 client, so boto3 signs requests as us-east-1.

If your Garage has a different s3_region (the sample config uses garage), Garage rejects HEAD requests like HeadBucket, which check_services makes, with a 400. HEAD responses have no body, so all you see is "Bad Request".

Listing and uploads still work with the wrong region, which makes it confusing to debug with aws s3 ls.

You can confirm with:

AWS_DEFAULT_REGION=us-east-1 aws --endpoint-url http://192.168.200.41 s3api head-bucket --bucket misau

If that returns a 400, add these to the config dict of S3_BLOB_DB_SETTINGS in localsettings.py, using the s3_region value from your garage.toml:

'config': {
   'signature_version': 's3v4',
   'region_name': 'garage',
   # Garage bug with multipart uploads (files over 8 MB),
   # see https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1228
   'request_checksum_calculation': 'when_required',
   'response_checksum_validation': 'when_required'
},

The last two lines are for a separate Garage issue: without them, files over 8 MB upload fine but fail to download with a checksum error.

Then restart HQ services and rerun check_services.

commcare-cloud regenerates localsettings.py on every update-config, and public.yml has no option for these yet. So you'll need to make this change on every machine running HQ.

We're looking at adding it to commcare-cloud so it sticks.

Let me know how this goes.