Users unable to sync or update application

We have reports from users that they are unable to sync their applications or download updates to the latest version of deployed application:

Latest code and formplayer releases as of 25-10-2022 have been applied to the server but there is no change. All services are up and reporting no errors:

Please assist

Hello, @nikotelhe.

Would it be possible to get a screenshot maybe of the error details (the user can select the "See Error Details" popup)?

Kind regards

Hi Charl, as below:

Thanks, @nikotelhe .

Is this something that only a few users experience, or more than a few (i.e. is this a wide-spread issue)?

Also, what happens when the users who are experiencing this issue clear their user data and try again? (Attention! They should not clear their data if they have any forms stored on their devices that have not been synced to the server yet!)

@CharlSmit
Yes its a widespread issue affecting users even after clearing their data.

New installations are also facing the same issue

Hi @nikotelhe

I believe the error "Global resource table was not ready for upgrading" can be caused by corrupt multimedia when syncing. So I'd like to check whether you are still experiencing this problem, or whether it was resolved when the multimedia issue was resolved. (Unable to install new application due to multimedia issue - #10 by Sravan_Reddy)

Hi @Norman_Hooper
Yes, this issue (Unable to install new application due to multimedia issue - #10 by Sravan_Reddy ) was initially resolved and users were able to install new application versions.

However, the current sync and update installation issue we are facing is yet to be resolved.

Hi @nikotelhe

When the users face an issue on the mobile sync or installation, it could be either due to the server or the mobile or a combination of both.

To help identify this, can you please check if there are any error logs on the web server when the users try to sync/install and provide them here.

FYI - the error logs are located in the webworker machine's /home/cchq/www/<env-name>/logs. You might need to tail the django log file in that location as you try to perform sync/install to obtain the error traceback.

Secondly, we have observed that we are able to install the application now. Can you please check if you are able to install the application now?

Thanks
Sravan


This is the page after clicking more details

Hi @Sravan_Reddy

Yes, users are now able to install the latest available application on the server however when they try to sync or start a supervision they are presented with the error: "unable to find lookup table jr://fixture/locations......

a tail of the log while attempting to sync is as follows:

couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 08:48:46,811 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 08:48:46,999 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 08:48:47,154 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 08:48:47,308 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 08:52:05,708 INFO [root] Processing form ef64ed1e-b959-4aad-b978-e4f2f2b4e029 as a device log
2022-11-04 08:52:08,849 INFO [root] Processing form 1b59467d-d51f-413b-b8d7-036eeea73450 as a device log
2022-11-04 08:52:59,966 INFO [root] Received Form d73f4131-8eec-476f-85c9-8b80db2ad86a with 0 attachments
2022-11-04 08:53:00,278 INFO [root] Finished normal processing for Form d73f4131-8eec-476f-85c9-8b80db2ad86a
2022-11-04 08:53:00,634 INFO [root] Processing form dc25e0cd-10ff-4ec7-bbfc-0a8aa142e249 as a device log
2022-11-04 08:53:00,823 INFO [root] Processing form a17f6df3-7d5b-4717-9fc2-6ff69180abc2 as a device log
2022-11-04 08:53:44,933 INFO [root] Received Form f2765dad-bf0b-43f9-b7b2-23c035107e96 with 0 attachments
2022-11-04 08:53:45,049 INFO [root] Finished normal processing for Form f2765dad-bf0b-43f9-b7b2-23c035107e96
2022-11-04 08:53:45,457 INFO [root] Processing form 4357cc9f-0aaa-4861-993c-128a364f4bd1 as a device log
2022-11-04 09:01:55,209 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/79df1fdeba8589f0859302cd04b922d1/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': '79df1fdeba8589f0859302cd04b922d1', 'key': 'cbc80afa3e334823a1f228761d5b2f63'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage 79df1fdeba8589f0859302cd04b922d1 attachment: 'da03264f2bee45696170a158c5a1ef56-437x437.PNG'
2022-11-04 09:01:56,381 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/79df1fdeba8589f0859302cd04b922d1/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': '79df1fdeba8589f0859302cd04b922d1', 'key': 'cbc80afa3e334823a1f228761d5b2f63'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage 79df1fdeba8589f0859302cd04b922d1 attachment: 'da03264f2bee45696170a158c5a1ef56-437x437.PNG'
2022-11-04 09:02:59,085 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/79df1fdeba8589f0859302cd04b922d1/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': '79df1fdeba8589f0859302cd04b922d1', 'key': 'cbc80afa3e334823a1f228761d5b2f63'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage 79df1fdeba8589f0859302cd04b922d1 attachment: 'da03264f2bee45696170a158c5a1ef56-437x437.PNG'
2022-11-04 09:04:13,446 INFO [root] Processing form f94f682b-dc5c-4504-8dfc-085c86062e7b as a device log
/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/PIL/Image.py:945: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
2022-11-04 09:07:10,974 INFO [root] Received Form 63f2bd80-cf24-4af7-a659-fd2c358bd0d4 with 0 attachments
2022-11-04 09:07:11,157 INFO [root] Finished normal processing for Form 63f2bd80-cf24-4af7-a659-fd2c358bd0d4
2022-11-04 09:07:12,239 INFO [root] Processing form 11cb903f-359e-44cf-b816-f540015a6950 as a device log
2022-11-04 09:14:03,430 INFO [root] Received Form f9a0bc5b-e5e5-48a5-80b0-eb4c249bea23 with 0 attachments
2022-11-04 09:14:03,775 INFO [root] Finished normal processing for Form f9a0bc5b-e5e5-48a5-80b0-eb4c249bea23
2022-11-04 09:14:04,227 INFO [root] Processing form aa537302-82a9-4051-982d-564bf48d4f43 as a device log
2022-11-04 09:14:47,354 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 09:14:47,541 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 09:14:47,733 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
2022-11-04 09:14:47,929 ERROR [django.request] Internal Server Error: /hq/multimedia/file/CommCareImage/e32329a2e840c72af35b9ee8b1d949de/green_show-gks3tp.png
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 191, in fetch_attachment
meta = db.metadb.get(parent_id=self._id, key=key)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/metadata.py", line 188, in get
raise BlobMeta.DoesNotExist(repr(kw))
corehq.blobs.models.BlobMeta.DoesNotExist: {'parent_id': 'e32329a2e840c72af35b9ee8b1d949de', 'key': 'commcarehq/e32329a2e840c72af35b9ee8b1d949de/5PhhXCCKM-Y'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/middleware.py", line 234, in inner
response = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 959, in dispatch
return super(ViewMultimediaFile, self).dispatch(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/views.py", line 993, in get
data, content_type = self.multimedia.get_display_file()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/apps/hqmedia/models.py", line 200, in get_display_file
data = self.fetch_attachment(self.attachment_id, stream=True).read()
File "/home/cchq/www/monolith/releases/2022-11-03_14.13/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: CommCareImage e32329a2e840c72af35b9ee8b1d949de attachment: '7d81721510ad2beba22413db76ed670c-256x256.PNG'
/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/PIL/Image.py:945: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
2022-11-04 09:22:40,177 INFO [root] Received Form 2c10377f-3c94-4977-9c24-892d196dda7e with 0 attachments
2022-11-04 09:22:40,549 INFO [root] Finished normal processing for Form 2c10377f-3c94-4977-9c24-892d196dda7e
2022-11-04 09:22:40,999 INFO [root] Processing form ab282f54-293b-4169-95b7-f7b80d43d267 as a device log
/home/cchq/www/monolith/releases/2022-11-03_14.13/python_env-3.9/lib/python3.9/site-packages/PIL/Image.py:945: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images

Just to add to what @nikotelhe has already posted. I tried to install the app again using QR Code.
When I installed version 74, which is our current version, the mobile app could not allow me to login. The error was bad response from server.

When I tried to installed version 64 with QR CODE, the installation did not complete, the message was as per the screenshot below.

When I installed version 50 with QR CODE...the installation completed successfully and I was able to login. But when I accessed any form I got the error of " Unable to find look up table Jr://fixture/locations.

I hope this clarifies a bit the predicament we are in

Hi Niko and Daniel

The error message does indicate that some multimedia is missing. Have you already tried the solution you have had success with?.

Regarding the location fixtures error, it's most likely a application configuration error. It's most likely that the user may not have access to locations or that locations have not been setup correctly. Please refer to the documentation on this to make sure these are setup correctly.

Hi Sravan,
Apologies for going back and forth on this matter. However, please note that this is happening to new installation as well as new users.

When an installation is done after upgrading to version 74 which is the current version, users are unable to:

  1. sync with the server with error message "An unexpected issue occurred during sync. Please try again.
  2. When user try to access a form they encounter the error "Unable to find lookup table 'jr://fixture/locations'. Make sure it exists and this user has access to it"
  3. at times registered users are unable to login with a response "Bad server Response"

According to Dimagi, the location settings were done quite alright and all users created have been assigned a location.

I hope this clarify our predicament

Hi Daniel

Have you tried to force pull the multimedia when pulling from remote? My suggestion is to first try this. If you face any errors, please share the corresponding Django error log here.

If that also doesn't work you will have to reinsert multimedia by following the process here reinsert_multimedia.md · GitHub but I suggest you to try the above method first.

Hi Sravan, last time we didn't see any logos in the Multimedia Reference checker. All logo's showed an error. We then took new images of all logo's which we replaced with existing ones, which fixed the issue.

we tried to apply the same 'fix' we did last time, which was replacing the logo's one by one in the Multimedia reference checker, even though this time the logo's were displaying correctly (see screenshot above)

what is happening now is that when we click on the icon (see where the mouse is in the above screenshot) it shows an error page (see screenshot below). Whereas on CCHQ it would normally show the icon in a larger format.

I think you are right that the sync issue is still multimedia related but we are not sure what the solution is. We think its something that needs to be fixed on the backend this time.

Niko, can you apply the steps mentioned by Sravan and update this ticket on the outcome?

Hello Niko

Have you tried to force pull the multimedia when pulling from remote? My suggestion is to first try this. If you face any errors, please share the corresponding Django error log here.

Have you tried this yet? It's important to try this first as it's an easier option and any error that you may face will help us fix that so that you don't have to face this again. Note that this is a UI option in the form of a checkbox called 'Force pull all multimedia' when you are pulling from master.

Hi @Sravan_Reddy, apologies for the late feedback but as follows:

Have you tried to force pull the multimedia when pulling from remote?

it's not applicable to this server because both the dev and prod project spaces are on the same server.

If that also doesn't work you will have to reinsert multimedia by following the process here reinsert_multimedia.md · GitHub but I suggest you to try the above method first.

this is the result from the first part of the steps provided:

It appears all multimedia is present

Can you apply a Linux utility fsck on blobdb server? it checks filesystems for errors or outstanding issues