85. Removing websockets

sirajhassan · April 3, 2025, 11:37am

Hi Team,

We are facing error below while deploying the code after change 85 is applied:

TASK [deploy_hq : Migrate databases] *****************************************************************************************************************************
failed: [192.168.1.80] (item=migrate_multi --noinput) => {"ansible_loop_var": "item", "changed": true, "cmd": ["./manage.py", "migrate_multi", "--noinput"], "delta": "0:01:00.365164", "end": "2025-04-03 10:34:49.617263", "item": "migrate_multi --noinput", "msg": "non-zero return code", "rc": 1, "start": "2025-04-03 10:33:49.252099", "stderr": "Traceback (most recent call last):\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/util/django_migrations.py\", line 121, in _inner\n    return migration_fn(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/migrations/0008_sync_es_with_couch_webusers.py\", line 8, in sync_couch_webusers_with_es\n    call_command('sync_es_webusers')\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/management/commands/sync_es_webusers.py\", line 14, in handle\n    update_user_in_es(None, CouchUser.wrap_correctly(user_doc))\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/apps/users/models.py\", line 1423, in wrap_correctly\n    doc_type = source['doc_type']\nTypeError: 'NoneType' object is not subscriptable\n2025-04-03T10:34:23Z <Greenlet at 0x7f518ec3da40: migrate_db('default')> failed with TypeError", "stderr_lines": ["Traceback (most recent call last):", "  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards", "    self.code(from_state.apps, schema_editor)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/util/django_migrations.py\", line 121, in _inner", "    return migration_fn(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/migrations/0008_sync_es_with_couch_webusers.py\", line 8, in sync_couch_webusers_with_es", "    call_command('sync_es_webusers')", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/management/commands/sync_es_webusers.py\", line 14, in handle", "    update_user_in_es(None, CouchUser.wrap_correctly(user_doc))", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/apps/users/models.py\", line 1423, in wrap_correctly", "    doc_type = source['doc_type']", "TypeError: 'NoneType' object is not subscriptable", "2025-04-03T10:34:23Z <Greenlet at 0x7f518ec3da40: migrate_db('default')> failed with TypeError"], "stdout": "\nThe following databases will be migrated:\n * default\n * auditcare\n * p1\n * p2\n * p3\n * p4\n * p5\n * p6\n * p7\n * p8\n * proxy\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7f518ec3da40: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/util/django_migrations.py\", line 121, in _inner\n    return migration_fn(*args, **kwargs)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/migrations/0008_sync_es_with_couch_webusers.py\", line 8, in sync_couch_webusers_with_es\n    call_command('sync_es_webusers')\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/management/commands/sync_es_webusers.py\", line 14, in handle\n    update_user_in_es(None, CouchUser.wrap_correctly(user_doc))\n  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/apps/users/models.py\", line 1423, in wrap_correctly\n    doc_type = source['doc_type']\nTypeError: 'NoneType' object is not subscriptable", "stdout_lines": ["", "The following databases will be migrated:", " * default", " * auditcare", " * p1", " * p2", " * p3", " * p4", " * p5", " * p6", " * p7", " * p8", " * proxy", " * synclogs", "", "", "The following databases will be skipped:", " * ucr", "", "", "======================= Error During Migration =======================", "<Greenlet at 0x7f518ec3da40: migrate_db('default')>", "Traceback (most recent call last):", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle", "    job.get()", "  File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get", "  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise", "    raise value.with_traceback(tb)", "  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards", "    self.code(from_state.apps, schema_editor)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/util/django_migrations.py\", line 121, in _inner", "    return migration_fn(*args, **kwargs)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/migrations/0008_sync_es_with_couch_webusers.py\", line 8, in sync_couch_webusers_with_es", "    call_command('sync_es_webusers')", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/./corehq/ex-submodules/pillowtop/management/commands/sync_es_webusers.py\", line 14, in handle", "    update_user_in_es(None, CouchUser.wrap_correctly(user_doc))", "  File \"/home/cchq/www/echis/releases/2025-04-03_10.21/corehq/apps/users/models.py\", line 1423, in wrap_correctly", "    doc_type = source['doc_type']", "TypeError: 'NoneType' object is not subscriptable"]}

The following error is also appearing in xfom-pillow log:

xform-pillow

Thank you,

Jonahtang · April 7, 2025, 4:42pm

Hi Siraj,

Thanks for sharing the traceback. The error appears to be happening during the 0008_sync_es_with_couch_webusers migration. Based on the stack trace, it looks like CouchUser.wrap_correctly(user_doc) is being called with user_doc = None, which causes a TypeError when it tries to access source['doc_type'].

This likely means that one of the document IDs being fetched from CouchDB is returning None, which usually happens if the document has been deleted or if the ID is invalid.

To resolve this, you'll need to identify which document ID is triggering the error. Once you identify the problematic doc, you can inspect it in CouchDB and either correct or remove it.

To find the document ID that is returning None, you can run this script in the shell

from corehq.apps.users.models import CouchUser
from corehq.util.couch_helpers import paginate_view
results = paginate_view(
    CouchUser.get_db(), 'all_docs/by_doc_type',
    chunk_size=100, startkey=["WebUser"], endkey=["WebUser", {}],
    include_docs=True, reduce=False)
for result in results:
    if result['doc'] is None:
        print(f"Document with id '{result['id']}' is None. Result: {result}")

Here’s the key part of the traceback for reference:

  ...
  File "pillowtop/management/commands/sync_es_webusers.py", line 14, in handle
    update_user_in_es(None, CouchUser.wrap_correctly(user_doc))
  ...
  File "corehq/apps/users/models.py", line 1423, in wrap_correctly
    doc_type = source['doc_type']
TypeError: 'NoneType' object is not subscriptable

sirajhassan · April 8, 2025, 8:00pm

Hi @Jonahtang

This is the output of the script.
However, I couldn't get any document with these IDs in the CouchDB. Is there any way to delete them from django shell?

Document with id '1335bcdccdf29e7d376ac1a8c769f3f1' is None. Result: {'id': '1335bcdccdf29e7d376ac1a8c769f3f1', 'key': ['WebUser', '1335bcdccdf29e7d376ac1a8c769f3f1'], 'value': None, 'doc': None}
Document with id '1da18e54f5e609b1140f5a146f7e8717' is None. Result: {'id': '1da18e54f5e609b1140f5a146f7e8717', 'key': ['WebUser', '1da18e54f5e609b1140f5a146f7e8717'], 'value': None, 'doc': None}
Document with id '4355cd5ace91391bb4872bd692c5b4d4' is None. Result: {'id': '4355cd5ace91391bb4872bd692c5b4d4', 'key': ['WebUser', '4355cd5ace91391bb4872bd692c5b4d4'], 'value': None, 'doc': None}
Document with id '5fc762ebc22e3a21a45c4fd9b7429747' is None. Result: {'id': '5fc762ebc22e3a21a45c4fd9b7429747', 'key': ['WebUser', '5fc762ebc22e3a21a45c4fd9b7429747'], 'value': None, 'doc': None}
Document with id 'a6ab902fb5439514fb1ad3a6d4002097' is None. Result: {'id': 'a6ab902fb5439514fb1ad3a6d4002097', 'key': ['WebUser', 'a6ab902fb5439514fb1ad3a6d4002097'], 'value': None, 'doc': None}

Jonahtang · April 17, 2025, 11:24pm

@sirajhassan

For each document you want to delete, use its id and run:

from corehq.apps.users.models import CouchUser
CouchUser.get_db().delete_doc(id)

For example:
CouchUser.get_db().delete_doc("1335bcdccdf29e7d376ac1a8c769f3f1")

sirajhassan · April 21, 2025, 7:07am

Hi @Jonahtang

"ResouceNotFound" error appears when trying to delete the documents:

In [2]: CouchUser.get_db().delete_doc("1335bcdccdf29e7d376ac1a8c769f3f1")
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/www/echis/releases/2025-04-11_12.57/python_env/lib/python3.9/site-packages/couchdbkit/client.py:477, in Database.get_rev(self, docid)
    476 try:
--> 477     response.raise_for_status()
    478 except HTTPError as e:

File ~/www/echis/releases/2025-04-11_12.57/python_env/lib/python3.9/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
   1023 if http_error_msg:
-> 1024     raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Object Not Found for url: http://192.168.1.52:25984/commcarehq__users/1335bcdccdf29e7d376ac1a8c769f3f1

During handling of the above exception, another exception occurred:

ResourceNotFound                          Traceback (most recent call last)
Cell In[2], line 1
----> 1 CouchUser.get_db().delete_doc("1335bcdccdf29e7d376ac1a8c769f3f1")

File ~/www/echis/releases/2025-04-11_12.57/python_env/lib/python3.9/site-packages/couchdbkit/client.py:678, in Database.delete_doc(self, doc, **params)
    676 elif isinstance(doc1, six.string_types): # we get a docid
    677     couch_doc = Document(self.cloudant_database, doc1)
--> 678     couch_doc['_rev'] = self.get_rev(doc1)
    680 # manual request because cloudant library doesn't return result
    681 res = self._request_session.delete(
    682     couch_doc.document_url,
    683     params={"rev": couch_doc["_rev"]},
    684 )

File ~/www/echis/releases/2025-04-11_12.57/python_env/lib/python3.9/site-packages/couchdbkit/client.py:480, in Database.get_rev(self, docid)
    478 except HTTPError as e:
    479     if e.response.status_code == 404:
--> 480         raise ResourceNotFound
    481     raise
    482 # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

ResourceNotFound:

Regards,

Jonahtang · April 23, 2025, 3:27am

@sirajhassan
That's strange - do you get that same error for all the document ids you previously identified? And what if you run CouchUser.get("1335bcdccdf29e7d376ac1a8c769f3f1")?

If the document does not exist that may be an indication of data corruption. You can try to use the following management commands to attempt to detect and repair corruption in the database.

./manage.py corrupt_couch count-missing users --doc-type=WebUser
./manage.py corrupt_couch repair users --doc-type=WebUser

If corruption is detected, but the repair command is not successful, try running the repair command with --min-tries=100 (the default is 10)

sirajhassan · April 23, 2025, 8:40am

Hi @Jonahtang
Yes, I get " ResouceNotFound" error for the all (five) document ids.

And the output of "./manage.py corrupt_couch count-missing users --doc-type=WebUser --min-tries=100" is:

2025-04-23 08:06:35,453 INFO [corehq.apps.hqadmin.corrupt_couch] processing users
2025-04-23 08:06:39,884 INFO [corehq.apps.hqadmin.corrupt_couch]   WebUser: missing=0 tries=(avg: 100.0, max: 100)

sirajhassan · April 27, 2025, 3:45pm

Hi @Jonahtang

Now able to deploy the code after fake documents were created (with the document IDs).

Thank you very much for your kind support.