Blobdb Exception

sirajhassan · April 6, 2023, 8:11am

Hi,

Mobile users failed to sync and create new mobile worker users.

Detail message from Django log:

2023-04-06 07:46:49,151 ERROR [django.request] Internal Server Error: /a/fmoh-echis/receiver/secure/d5dc553505b54eafbce0eebd0f5820be/
Traceback (most recent call last):
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
return callback(request, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 649, in wrapped_view
return view_func(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/receiverwrapper/views.py", line 391, in secure_post
return decorated_view(request, domain, app_id=app_id)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/ota/utils.py", line 207, in _inner
response = f(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 262, in safe_fn_with_domain
return _inner(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 568, in _inner
return fn(request, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/auth.py", line 150, in wrapper
return view(request, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 465, in _inner
return fn(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 258, in _inner
return fn(request, domain, *args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/domain/decorators.py", line 387, in wrapped_view
return view_func(*args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/receiverwrapper/views.py", line 341, in _secure_post_basic
return _process_form(
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/apps/receiverwrapper/views.py", line 139, in _process_form
result = submission_post.run()
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/submission_post.py", line 345, in run
openrosa_kwargs['error_message'] = self.save_processed_models(case_db, xforms,
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/ddtrace/tracer.py", line 787, in func_wrapper
return f(*args, **kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/submission_post.py", line 426, in save_processed_models
self.interface.save_processed_models(
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/interfaces/processor.py", line 148, in save_processed_models
return self.processor.save_processed_models(
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/backends/sql/processor.py", line 115, in save_processed_models
XFormInstance.objects.save_new_form(processed_forms.submitted)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/models/forms.py", line 291, in save_new_form
attachment_writer.write()
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/models/attachment.py", line 226, in write
self.saved_attachments = [
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/models/attachment.py", line 227, in
attachment.write(self.blob_db, self.form)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/form_processor/models/attachment.py", line 134, in write
return blob_db.put(
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/blobs/atomic.py", line 28, in put
meta = self.db.put(content, **kw)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/corehq/blobs/s3db.py", line 89, in put
s3_bucket.upload_fileobj(content, meta.key, Callback=_track_transfer)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/boto3/s3/inject.py", line 678, in bucket_upload_fileobj
return self.meta.client.upload_fileobj(
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
return future.result()
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in call
return self._execute_main(kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/s3transfer/upload.py", line 758, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/cchq/www/echis/releases/2023-04-06_00.45/python_env-3.9/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Resource requested is unreadable, please reduce your request rate

We kindly request your feedback please.

Thanks,

demis08 · April 6, 2023, 10:45am

Hi Siraj,

EXCEPTION (Took   6.13s) blobdb         : ClientError('An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Resource requested is unreadable, please reduce your request rate')

it means the concurrent put operation is more than what object storage can handle, i would suggest to check minio cluster status.

make sure all minio cluster are operational first. can u paste the output of mc admin info minio

sirajhassan · April 6, 2023, 11:03am

Thank you @demis08 for your swift response.

The cluster looks good.

● minio1:9000
Uptime: 20 hours
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio2:9000
Uptime: 20 hours
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio3:9000
Uptime: 12 hours
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio4:9000
Uptime: 1 day
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio5:9000
Uptime: 1 day
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio6:9000
Uptime: 1 day
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio7:9000
Uptime: 1 day
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

● minio8:9000
Uptime: 1 day
Version: 2022-09-01T23:53:36Z
Network: 8/8 OK
Drives: 2/2 OK
Pool: 1

Pools:
1st, Erasure sets: 1, Disks per erasure set: 16

16 drives online, 0 drives offline

demis08 · April 6, 2023, 11:12am

so it's better to set rate limiting on based on eCHIS usage patterns and expected traffic. this will prevent excessive requests and ensure that minio-server can handle the expected load

demis08 · April 6, 2023, 11:22am

minio has builtin rate limiting feature. you can set with a command:

mc admin bucket quota minio/<bucket-name> --quota 100

This will set the rate limit to 100 requests per second.

Monitor the performance of your Minio server and adjust the rate limits as needed to ensure that your application remains within the specified limits.