Some Applications give error after recent upgrade from older version of CommCare

Having recently upgraded CommCareHQ from an older version, some Applications don't open and give an error when attempting to open them as shown in Figures 1 & 2. I can not seem to figure out which log files to look at what is going wrong.

Any ideas or suggestions would be appreciated

Figure 1. Clicking on the Application in the Applications pane results in Figure 2.

Figure 2. The view shown having attempted to open the Application

From looking at the log monolith-commcarehq.django.log, I managed to get the following which I think might be the error. Is there anyway of resolving this to get the Application to start or be editted?

ERROR 2023-05-10 19:41:50,914 log 11639 140708578756800 Internal Server Error: /a/hope-worldwide-sa/apps/view/cf2ecc6429d26597e2294d6dc5283e17/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/domain/decorators.py", line 111, in _inner
return call_view()
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/domain/decorators.py", line 84, in call_view
def call_view(): return view_func(req, domain_name, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/views/apps.py", line 701, in view_app
return view_generic(request, domain, app_id, release_manager=True)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/ex-submodules/dimagi/utils/couch/resource_conflict.py", line 20, in new_fn
return fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/views/view_generic.py", line 77, in view_generic
app = get_app(domain, app_id)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/dbaccessors.py", line 246, in get_app
return wrap_app(app, wrap_cls=wrap_cls)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/dbaccessors.py", line 154, in wrap_app
return cls.wrap(app_doc)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 4579, in wrap
self = super(Application, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 4108, in wrap
self = super(ApplicationBase, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 3847, in wrap
self = super(LazyBlobDoc, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/blobs/mixin.py", line 89, in wrap
return super(BlobMixin, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 3826, in init
super(LazyBlobDoc, self).init(*args, **kwargs)
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 24, in jsonobject.containers.JsonArray.init
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2254, in wrap
return Module.wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2264, in wrap
return super(ModuleBase, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2231, in init
super(ModuleBase, self).init(*args, **kwargs)
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 24, in jsonobject.containers.JsonArray.init
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 1062, in wrap
return super(FormBase, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 148, in jsonobject.containers.JsonDict.init
File "jsonobject/containers.pyx", line 155, in jsonobject.containers.JsonDict.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 196, in jsonobject.base.JsonObjectBase.init
File "jsonobject/utils.pyx", line 9, in jsonobject.utils.check_type
jsonobject.exceptions.BadValueError: JsonObject must wrap a dict or None. Found object of type: <class 'str'>
ERROR 2023-05-10 19:41:50,914 log 11639 140708578756800 Internal Server Error: /a/hope-worldwide-sa/apps/view/cf2ecc6429d26597e2294d6dc5283e17/
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/domain/decorators.py", line 111, in _inner
return call_view()
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/domain/decorators.py", line 84, in call_view
def call_view(): return view_func(req, domain_name, *args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/views/apps.py", line 701, in view_app
return view_generic(request, domain, app_id, release_manager=True)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/ex-submodules/dimagi/utils/couch/resource_conflict.py", line 20, in new_fn
return fn(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/views/view_generic.py", line 77, in view_generic
app = get_app(domain, app_id)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/dbaccessors.py", line 246, in get_app
return wrap_app(app, wrap_cls=wrap_cls)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/dbaccessors.py", line 154, in wrap_app
return cls.wrap(app_doc)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 4579, in wrap
self = super(Application, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 4108, in wrap
self = super(ApplicationBase, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 3847, in wrap
self = super(LazyBlobDoc, cls).wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/blobs/mixin.py", line 89, in wrap
return super(BlobMixin, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 3826, in init
super(LazyBlobDoc, self).init(*args, **kwargs)
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 24, in jsonobject.containers.JsonArray.init
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2254, in wrap
return Module.wrap(data)
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2264, in wrap
return super(ModuleBase, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 2231, in init
super(ModuleBase, self).init(*args, **kwargs)
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 24, in jsonobject.containers.JsonArray.init
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "/home/cchq/www/monolith/releases/2023-05-10_14.32/corehq/apps/app_manager/models.py", line 1062, in wrap
return super(FormBase, cls).wrap(data)
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.init
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 180, in jsonobject.base_properties.JsonContainerProperty.wrap
File "jsonobject/containers.pyx", line 148, in jsonobject.containers.JsonDict.init
File "jsonobject/containers.pyx", line 155, in jsonobject.containers.JsonDict.__wrap
File "jsonobject/properties.pyx", line 154, in jsonobject.properties.ObjectProperty.wrap
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 196, in jsonobject.base.JsonObjectBase.init
File "jsonobject/utils.pyx", line 9, in jsonobject.utils.check_type
jsonobject.exceptions.BadValueError: JsonObject must wrap a dict or None. Found object of type: <class 'str'>

It could be that a migration was missed or failed somehow. If you're hooked up to Sentry it would be useful to get additional context on the error, specifically, the data that was passed to super(FormBase, cls).wrap(data).

There was a migration run last year which should have run automatically during the deploy but we could try running again.

You'll need to create a private release with a previous version:

cchq monolith deploy --private --commcare-rev=3c47a08dad06c20f376b25ce2bcd4f307ff5f6e6

Then run the migrate_to_conditional_case_update command:

cchq monolith django-manage --release <name of private release> migrate_to_conditional_case_update

# and again for deleted apps

cchq monolith django-manage --release <name of private release> migrate_to_conditional_case_update --deleted-apps-only

So I am getting a message that these have been run already. Is there anything else I can try?

(monolith) ubuntu@monolith:/$ cchq monolith django-manage --release 2023-05-11_08.26 migrate_to_conditional_case_update
ssh ubuntu@172.31.22.241 -t -o UserKnownHostsFile=/home/ubuntu/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2023-05-11_08.26; python_env/bin/python manage.py migrate_to_conditional_case_update'"'"''
Ubuntu 18.04.5 LTS
2023-05-11 08:35:55,529 INFO [app_migration] This migration command has already been run on this environment. Exiting...
Connection to 172.31.22.241 closed.
(monolith) ubuntu@monolith:/$ cchq monolith django-manage --release 2023-05-11_08.26 migrate_to_conditional_case_update --deleted-apps-only       ssh ubuntu@172.31.22.241 -t -o UserKnownHostsFile=/home/ubuntu/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2023-05-11_08.26; python_env/bin/python manage.py migrate_to_conditional_case_update --deleted-apps-only'"'"''
Ubuntu 18.04.5 LTS
2023-05-11 08:36:32,457 INFO [app_migration] This migration command has already been run on this environment. Exiting...
Connection to 172.31.22.241 closed.

I don't know about Sentry, but looking at the commcare-hq.gunicorn.log I am also seeing many of these errors:

ConnectionRefusedError: [Errno 111] Connection refused
[2023-05-11 08:38:49 +0000] [24253] [DEBUG] Closing connection.
[2023-05-11 08:38:49 +0000] [19798] [DEBUG] POST /a/lego-ltp/cloudcare/apps/report_formplayer_error
[2023-05-11 08:38:49 +0000] [19798] [WARNING] Error sending message to statsd
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/gunicorn/instrument/statsd.py", line 127, in _sock_send
    self.sock.send(msg)
ConnectionRefusedError: [Errno 111] Connection refused
[2023-05-11 08:38:49 +0000] [19798] [DEBUG] Closing connection.
[2023-05-11 08:39:00 +0000] [24253] [DEBUG] POST /a/hope-worldwide-sa/receiver/secure/8b583baae209421b842dd4f2d3bf01a5/
[2023-05-11 08:39:00 +0000] [24253] [WARNING] Error sending message to statsd
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2023-05-10_14.32/python_env-3.9/lib/python3.9/site-packages/gunicorn/instrument/statsd.py", line 127, in _sock_send
    self.sock.send(msg)
ConnectionRefusedError: [Errno 111] Connection refused

I can't see too many other errors appearing besides the original

You shouldn't need to worrry about the statsd errors. I think that just means it's not running / installed on the machine but it's not necessary and shouldn't cause anything to fail.

Could you try using this function in a shell to get the problem form?

import json
from corehq.apps.app_manager.models import *

def find_problem_form(app_id: str):
    app = Application.get_db().get(app_id)
    for module in app["modules"]:
        for form in module["forms"]:
            try:
                FormBase.wrap(form)
            except Exception:
                print(json.dumps(form))

find_problem_form("cf2ecc6429d26597e2294d6dc5283e17")

Thank you very much,

How should I run it in a shell? I first tried running it from the command line as a temporary file, and I got an error regarding ModuleNotFoundError: No module named 'corehq.app' so I changed line 2 to from corehq.apps.app_manager.models import * , but then I got a ModuleNotFoundError: No module named 'dimagi' error.

Dirk

Sorry, I meant a Django shell:

cchq monolith django-manage shell

Then you can paste in the code block (thanks for the fix).

Ok! thank you! There is a significant response - 34 Messages which all look like the following:

{"name": {"en": "Register a Centre"}, "form_type": "module_form", "form_filter": "#user/hen = 'yes'", "requires": "none", "actions": {"open_case": {"condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "name_path": "/data/what_is_the_full_name_of_this_centre", "external_id": null, "doc_type": "OpenCaseAction", "name_update": {"question_path": null, "update_mode": "always", "doc_type": "ConditionalCaseUpdate"}}, "update_case": {"update": {"what_is_the_full_name_of_this_centre": "/data/what_is_the_full_name_of_this_centre", "which_province_is_this": "/data/which_province_is_this", "which_site_is_this": "/data/which_site_is_this", "what_is_the_name_of_the_suburb": "/data/what_is_the_name_of_the_suburb", "what_is_the_address_of_the_centre": "/data/what_is_the_address_of_the_centre", "what_is_the_name_and_surname_of_the_principal": "/data/what_is_the_name_and_surname_of_the_principal", "does_this_person_have_an_id_number": "/data/does_this_person_have_an_id_number", "what_is_the_id_number": "/data/what_is_the_id_number", "what_is_the_principals_home_address": "/data/what_is_the_principals_home_address", "what_is_the_work_phone_number": "/data/what_is_the_work_phone_number", "what_is_the_first_cellphone_number": "/data/what_is_the_first_cellphone_number", "what_is_the_second_cellphone_number": "/data/what_is_the_second_cellphone_number", "what_is_the_e-mail_address_of_the_centre_or_principal": "/data/what_is_the_e-mail_address_of_the_centre_or_principal", "how_many_children_are_in_the_ecd_centre": "/data/how_many_children_are_in_the_ecd_centre", "how_much_do_you_charge_for_fees_per_month": "/data/how_much_do_you_charge_for_fees_per_month", "have_the_principal_and_management_signed_the_child_safeguarding_policy": "/data/have_the_principal_and_management_signed_the_child_safeguarding_policy", "has_this_ecd_centre_paid_the_once-off_registration_fee": "/data/has_this_ecd_centre_paid_the_once-off_registration_fee", "has_this_ecd_centre_paid_the_annual_membership_fee": "/data/has_this_ecd_centre_paid_the_annual_membership_fee", "did_this_ecd_centre_pay_for_a_membership_t-shirt": "/data/did_this_ecd_centre_pay_for_a_membership_t-shirt", "centre_case_id": "/data/centre_case_id", "what_is_the_date_the_centre_will_belong_to_the_hen_network_from_leave_blank": "/data/what_is_the_date_the_centre_will_belong_to_the_hen_network_from_leave_blank"}, "condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "UpdateCaseAction"}, "close_case": {"condition": {"type": "never", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "FormAction"}, "case_preload": {"preload": {}, "condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "PreloadAction"}, "usercase_update": {"update": {"what_is_the_address_of_the_centre": "/data/what_is_the_address_of_the_centre"}, "condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "UpdateCaseAction"}, "usercase_preload": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "subcases": [], "open_referral": {"name_path": null, "doc_type": "OpenReferralAction", "followup_date": null, "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "update_referral": {"followup_date": null, "doc_type": "UpdateReferralAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "close_referral": {"condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}, "doc_type": "FormAction"}, "referral_preload": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "load_from_form": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "doc_type": "FormActions"}, "doc_type": "Form", "unique_id": "6dfa0c1b32e84c21b21698b87167fa7c", "show_count": false, "xmlns": "http://openrosa.org/formdesigner/5A08A2BA-5F23-48C7-A836-F358199D6868", "version": null, "post_form_workflow": "previous_screen", "post_form_workflow_fallback": "default", "auto_gps_capture": false, "form_links": [], "schedule_form_id": null, "custom_assertions": [], "custom_instances": [], "case_references_data": {"load": {}, "save": {}, "doc_type": "CaseReferences"}, "is_release_notes_form": false, "enable_release_notes": false, "session_endpoint_id": null, "function_datum_endpoints": [], "comment": "", "media_image": {"en": "jr://file/commcare/image/module0_form0_en.png"}, "media_audio": {"en": ""}, "custom_icons": [], "use_default_image_for_all": false, "use_default_audio_for_all": false}

...

{"name": {"en": "Before & After Pictures"}, "form_type": "module_form", "requires": "case", "actions": {"open_case": {"condition": {"type": "never", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "name_path": null, "external_id": null, "doc_type": "OpenCaseAction", "name_update": {"question_path": null, "update_mode": "always", "doc_type": "ConditionalCaseUpdate"}}, "update_case": {"update": {"are_you_taking_before_or_after_pictures": "/data/are_you_taking_before_or_after_pictures"}, "condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "UpdateCaseAction"}, "close_case": {"condition": {"type": "never", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "FormAction"}, "case_preload": {"preload": {}, "condition": {"type": "always", "question": null, "answer": null, "operator": null, "doc_type": "FormActionCondition"}, "doc_type": "PreloadAction"}, "usercase_update": {"update": {}, "doc_type": "UpdateCaseAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "usercase_preload": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "subcases": [], "open_referral": {"name_path": null, "doc_type": "OpenReferralAction", "followup_date": null, "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "update_referral": {"followup_date": null, "doc_type": "UpdateReferralAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "close_referral": {"condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}, "doc_type": "FormAction"}, "referral_preload": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "load_from_form": {"preload": {}, "doc_type": "PreloadAction", "condition": {"type": "never", "question": null, "answer": null, "operator": "=", "doc_type": "FormActionCondition"}}, "doc_type": "FormActions"}, "doc_type": "Form", "unique_id": "819b60cb6ea440d4b626fcba6c8b9072", "show_count": false, "xmlns": "http://openrosa.org/formdesigner/FDF4425E-0BEA-4CE0-8346-CCD1D425AC8D", "version": null, "post_form_workflow": "root", "post_form_workflow_fallback": "", "auto_gps_capture": false, "form_links": [], "schedule_form_id": null, "custom_assertions": [], "custom_instances": [], "case_references_data": {"load": {"/data/please_take_a_photo_of_the_area_to_be_refurbished": ["#case/will_this_centre_receive_infrastructure_support"], "/data/please_take_a_photo_of_the_area_that_has_been_refurbished": ["#case/will_this_centre_receive_infrastructure_support"], "/data/please_take_a_photo_of_the_area_to_be_equipped": ["#case/will_this_centre_receive_equipment_support"], "/data/please_take_a_photo_of_the_area_that_has_been_equipped": ["#case/will_this_centre_receive_equipment_support"]}, "save": {}, "doc_type": "CaseReferences"}, "is_release_notes_form": false, "enable_release_notes": false, "session_endpoint_id": null, "function_datum_endpoints": [], "comment": "", "media_image": {"en": "jr://file/commcare/image/module20_form2_en.png"}, "media_audio": {"en": ""}, "custom_icons": [], "use_default_image_for_all": false, "use_default_audio_for_all": false}

What can I look for or should do with these?

Thanks Dirk. I did some testing on my side and I have a hypothesis

The migrate_to_conditional_case_update command did run (as we've seen) but it only ran after other changes were made which prevented it from doing what it was supposed to do.

So I think you should re-run the command as described in my previous comment but add the following command line flags: --force-run-again --start-from-scratch.

Note that this must be run from an earlier code version (3c47a08dad06c20f376b25ce2bcd4f307ff5f6e6), not the latest release.

Brilliant! Thanks Simon

I first ran as follows:

(monolith) ubuntu@monolith:~$ cchq monolith django-manage --release 2023-05-11_08.26 migrate_to_conditional_case_update --force-run-again --start-from-scratch
ssh ubuntu@172.31.22.241 -t -o UserKnownHostsFile=/home/ubuntu/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2023-05-11_08.26; python_env/bin/python manage.py migrate_to_conditional_case_update --force-run-again --start-from-scratch'"'"''
Ubuntu 18.04.5 LTS
2023-05-11 13:34:58,909 INFO [app_migration] migrating 1 apps in dirk-proj1
couldn't find 0 docs
ignored 1 docs
deleted 0 docs
updated 0 docs
...
2023-05-11 13:37:14,783 INFO [app_migration] migrating 308 apps in lego-ltp
couldn't find 0 docs
ignored 0 docs
deleted 0 docs
updated 308 docs
2023-05-11 13:38:08,429 INFO [app_migration] Migrated domain #7
2023-05-11 13:38:08,430 INFO [app_migration] Completed in 0:03:09.541727.
Connection to 172.31.22.241 closed.
(monolith) ubuntu@monolith:~$

which which went through great, but then when I ran the 2nd command (with the --deleted-apps-only), it crashed as follows:

(monolith) ubuntu@monolith:~$ cchq monolith django-manage --release 2023-05-11_08.26 migrate_to_conditional_case_update --deleted-apps-only --force-run-again --start-from-scratch
ssh ubuntu@172.31.22.241 -t -o UserKnownHostsFile=/home/ubuntu/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2023-05-11_08.26; python_env/bin/python manage.py migrate_to_conditional_case_update --deleted-apps-only --force-run-again --start-from-scratch'"'"''
Ubuntu 18.04.5 LTS
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2023-05-11_08.26/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/cchq/www/monolith/releases/2023-05-11_08.26/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/cchq/www/monolith/releases/2023-05-11_08.26/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:
...
During handling of the above exception, another exception occurred:
...
  File "/home/cchq/www/monolith/releases/2023-05-11_08.26/python_env/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Connection to 172.31.22.241 closed.
(monolith) ubuntu@monolith:~$

At this point couchdb was down, so after a

commcare-cloud monolith downtime start
sudo service couchdb stop
sudo service couchdb start
commcare-cloud monolith downtime end

All the services came back online, and the Applications now work!

Thank you so much!

That's great! I'm glad everything is working again!

1 Like
I'd recommend to check some basic points. Confirm if the form with the unique ID '2b5ad0263e267741a1a9b8e7781761a3bb485f4d' is accessible in the upstream application. Ensure that the form exists and is properly linked to the application. If it is not accessible, you may need to investigate why the form is missing or not properly linked. Review server configuration. Examine the server configuration for any possible discrepancies between the local server and the staging server. Rebuild or relink the application. Make sure to follow proper procedures and backup any necessary data before performing this action.

found project