Thanks @Ethan_Soergel this is what I get when clicking virtually any application in the browser:
2022-03-11 09:08:05,748 ERROR [django.request] Internal Server Error: /a/xxxxxxxx-dev/apps/view/cbc4862d14324090845f87a413d8d82a/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/blobs/mixin.py", line 192, in fetch_attachment
blob = meta.open()
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/blobs/models.py", line 111, in open
return db.get(meta=self)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/blobs/fsdb.py", line 68, in get
raise NotFound(key)
corehq.blobs.exceptions.NotFound: 2598d588accb4f89920d7892198fd53a
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/python_env-3.9/lib/python3.9/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/domain/decorators.py", line 115, in _inner
return call_view()
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/domain/decorators.py", line 85, in call_view
def call_view(): return view_func(req, domain_name, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/views/apps.py", line 692, in view_app
return view_generic(request, domain, app_id, release_manager=True)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/ex-submodules/dimagi/utils/couch/resource_conflict.py", line 20, in new_fn
return fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/views/view_generic.py", line 163, in view_generic
context.update(get_releases_context(request, domain, app_id))
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/views/releases.py", line 222, in get_releases_context
multimedia_state = app.check_media_state()
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/hqmedia/models.py", line 997, in check_media_state
for media in self.all_media():
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/python_env-3.9/lib/python3.9/site-packages/memoized.py", line 20, in _memoized
cache[key] = value = fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/hqmedia/models.py", line 787, in all_media
form.validate_form()
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/models.py", line 1181, in validate_form
form = self.wrapped_xform()
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/models.py", line 1173, in wrapped_xform
return XForm(self.source, domain=self.get_app().domain)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/models.py", line 790, in __get__
source = app.lazy_fetch_attachment(filename)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/apps/app_manager/models.py", line 4022, in lazy_fetch_attachment
content = self.fetch_attachment(name)
File "/home/cchq/www/monolith/releases/2022-03-08_18.52/corehq/blobs/mixin.py", line 194, in fetch_attachment
raise ResourceNotFound(
couchdbkit.exceptions.ResourceNotFound: Application cbc4862d14324090845f87a413d8d82a attachment: '81c3984c748946e0b60afc760dc324bd.xml'
Is it possible something went awry with my recent server upgrade?
If I look at my /opt/blobdb dir, It seems 2598d588accb4f89920d7892198fd53a does indeed exist... going to investigate further
Perhaps this is a rights issue?
-rw-r--r-- 1 cchq cchq 44537 Feb 16 13:20 /opt/data/blobdb/2598d588accb4f89920d7892198fd53a