Installation Commcare Broken Help needed

I am trying install commcare but its having serious issues with redis been available. This is a new install Ubuntu 22.04.

commcare-cloud coronams deploy-stack -e 'CCHQ_IS_FRESH_INSTALL=1' --skip-check --branch=master --skip-tags=users --quiet -vvv

fatal: [172.16.2.72]: FAILED! => {
    "changed": false,
    "cmd": [
        "./manage.py",
        "migrate_multi",
        "--noinput"
    ],
    "invocation": {
        "module_args": {
            "ack_venv_creation_deprecation": null,
            "app_path": "/home/cchq/www/monolith/current",
            "apps": null,
            "cache_table": null,
            "clear": false,
            "command": "migrate_multi --noinput",
            "database": null,
            "failfast": false,
            "fixtures": null,
            "link": null,
            "merge": null,
            "project_path": "/home/cchq/www/monolith/current",
            "pythonpath": null,
            "settings": null,
            "skip": null,
            "testrunner": null,
            "virtualenv": "/home/cchq/www/monolith/current/python_env"
        }
    },
    "msg": "stdout: \nThe following databases will be migrated:\n * default\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7ff5b44f1e00: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django_redis/cache.py\", line 29, in _decorator\n    return method(self, *args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django_redis/cache.py\", line 104, in delete\n    result = self.client.delete(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django_redis/client/default.py\", line 375, in delete\n    raise ConnectionInterrupted(connection=client) from e\ndjango_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error 111 connecting to localhost:6379. Connection refused.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/accounting/migrations/0001_squashed_0052_ensure_report_builder_plans.py\", line 29, in _cchq_software_plan_bootstrap\n    ensure_plans(pricing_config, verbose=True, apps=apps)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/accounting/bootstrap/utils.py\", line 39, in ensure_plans\n    _clear_cache(SoftwarePlan.objects.all(), DefaultProductPlan.objects.all())\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/accounting/bootstrap/utils.py\", line 241, in _clear_cache\n    SoftwarePlan.get_version.clear(software_plan)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/quickcache/quickcache_helper.py\", line 96, in clear\n    self.cache.delete(key)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/quickcache/cache_helpers.py\", line 86, in delete\n    cache.delete(key)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/quickcache/cache_helpers.py\", line 39, in delete\n    return self.cache.delete(self.prefixed_key(key))\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django_redis/cache.py\", line 36, in _decorator\n    raise e.__cause__\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django_redis/client/default.py\", line 373, in delete\n    return client.delete(self.make_key(key, version=version, prefix=prefix))\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/redis/commands/core.py\", line 1673, in delete\n    return self.execute_command(\"DEL\", *names)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/redis/client.py\", line 1255, in execute_command\n    conn = self.connection or pool.get_connection(command_name, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/redis/connection.py\", line 1442, in get_connection\n    connection.connect()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/redis/connection.py\", line 704, in connect\n    raise ConnectionError(self._error_message(e))\nredis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.\n\n\n:stderr: System check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?:

Error 111 connecting to localhost:6379. Connection refused.\n2025-03-22T15:48:47Z <Greenlet at 0x7f180a6b1e00: migrate_db('default')> failed with ConnectionError\n\n",
    "path": "/home/cchq/www/monolith/current/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin",
    "syspath": [
        "/tmp/ansible_django_manage_payload_sr_grr2u/ansible_django_manage_payload.zip",
        "/usr/lib/python310.zip",
        "/usr/lib/python3.10",
        "/usr/lib/python3.10/lib-dynload",
        "/usr/local/lib/python3.10/dist-packages",
        "/usr/lib/python3/dist-packages"
    ]
}


Hi @alhamdu1

It appears redis isn't reachable on your setup.

You can check if the services are running okay by running the command shared on the installation steps here,

I did check the Installation scripts did not install redis.

I went through the troubleshooting, that did not work also.

I install redis on the worker manually, the redis issue passed, but now the django migration is not working.

    "changed": false,
    "cmd": [
        "./manage.py",
        "migrate_multi",
        "--noinput"
    ],
    "invocation": {
        "module_args": {
            "ack_venv_creation_deprecation": null,
            "app_path": "/home/cchq/www/monolith/current",
            "apps": null,
            "cache_table": null,
            "clear": false,
            "command": "migrate_multi --noinput",
            "database": null,
            "failfast": false,
            "fixtures": null,
            "link": null,
            "merge": null,
            "project_path": "/home/cchq/www/monolith/current",
            "pythonpath": null,
            "settings": null,
            "skip": null,
            "testrunner": null,
            "virtualenv": "/home/cchq/www/monolith/current/python_env"
        }
    },
    "msg": "stdout: \nThe following databases will be migrated:\n * default\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7f546d107e00: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n    conn = connection.create_connection(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n    raise err\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n    sock.connect(sa)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n    self._internal_connect(address)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n    raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n    super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n  File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n    self.connect()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n    conn = self._new_conn()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n    retries = retries.increment(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n    raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n    get_preindex_plugin('userreports').sync_design_docs()\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n    for design in self.get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n    for design in self._get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n    db = Database(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n    self.cloudant_database.create()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n    if not throw_on_exists and self.exists():\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n    resp = self.r_session.head(self.database_url)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n    return self.request(\"HEAD\", url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n    return super(BasicSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n    resp = real_request(self, method, url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n    resp = super(ClientSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\n\n:stderr: System check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n    conn = connection.create_connection(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n    raise err\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n    sock.connect(sa)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n    self._internal_connect(address)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n    raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n    super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n  File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n    self.connect()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n    conn = self._new_conn()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n    retries = retries.increment(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n    raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n    get_preindex_plugin('userreports').sync_design_docs()\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n    for design in self.get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n    for design in self._get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n    db = Database(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n    self.cloudant_database.create()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n    if not throw_on_exists and self.exists():\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n    resp = self.r_session.head(self.database_url)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n    return self.request(\"HEAD\", url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n    return super(BasicSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n    resp = real_request(self, method, url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n    resp = super(ClientSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f546b7c27c0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n2025-03-24T09:57:38Z <Greenlet at 0x7f546d107e00: migrate_db('default')> failed with ConnectionError\n\n",
    "path": "/home/cchq/www/monolith/current/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin",
    "syspath": [
        "/tmp/ansible_django_manage_payload_zd3nxrkh/ansible_django_manage_payload.zip",
        "/usr/lib/python310.zip",
        "/usr/lib/python3.10",
        "/usr/lib/python3.10/lib-dynload",
        "/usr/local/lib/python3.10/dist-packages",
        "/usr/lib/python3/dist-packages"
    ]
} ```

Hi @alhamdu1

I did check the Installation scripts did not install redis.

Redis should be getting installed if it's setup to. Installing things directly might result in them not getting configured correctly and so, they should be setup through commcare cloud only.

It seems you are now having trouble with couch db being unreachable.

Running the check_services command I shared earlier will give you a status of all services.

Your right, manual install will break stuff, and I get it. I have removed manual redis and ran the check_services

ssh abello@172.16.2.72 -t -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.5 LTS
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.
FAILURE (Took   0.10s) kafka          : No Kafka topics found
EXCEPTION (Took   0.00s) redis          : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
FAILURE (Took   0.02s) postgres       : default:commcarehq:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK No users found in postgres
EXCEPTION (Took   0.00s) couch          : Service check errored with exception 'ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__apps (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d11928a00>: Failed to establish a new connection: [Errno 111] Connection refused'))"))'
EXCEPTION (Took   0.00s) celery         : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
SUCCESS (Took   0.05s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.01s) blobdb         : Successfully saved a file to the blobdb
FAILURE (Took   5.17s) formplayer     : Could not establish a connection in time https://commcarehq.coronams.com/formplayer/serverup
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK
Connection to 172.16.2.72 closed.

Also re-ran the install back again


fatal: [172.16.2.72]: FAILED! => {
    "changed": false,
    "cmd": [
        "./manage.py",
        "migrate_multi",
        "--noinput"
    ],
    "invocation": {
        "module_args": {
            "ack_venv_creation_deprecation": null,
            "app_path": "/home/cchq/www/monolith/current",
            "apps": null,
            "cache_table": null,
            "clear": false,
            "command": "migrate_multi --noinput",
            "database": null,
            "failfast": false,
            "fixtures": null,
            "link": null,
            "merge": null,
            "project_path": "/home/cchq/www/monolith/current",
            "pythonpath": null,
            "settings": null,
            "skip": null,
            "testrunner": null,
            "virtualenv": "/home/cchq/www/monolith/current/python_env"
        }
    },
    "msg": "stdout: \nThe following databases will be migrated:\n * default\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7ff59d19fe00: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n    conn = connection.create_connection(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n    raise err\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n    sock.connect(sa)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n    self._internal_connect(address)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n    raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n    super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n  File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n    self.connect()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n    conn = self._new_conn()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n    retries = retries.increment(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n    raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n    get_preindex_plugin('userreports').sync_design_docs()\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n    for design in self.get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n    for design in self._get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n    db = Database(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n    self.cloudant_database.create()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n    if not throw_on_exists and self.exists():\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n    resp = self.r_session.head(self.database_url)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n    return self.request(\"HEAD\", url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n    return super(BasicSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n    resp = real_request(self, method, url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n    resp = super(ClientSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\n\n:stderr: System check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n    conn = connection.create_connection(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n    raise err\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n    sock.connect(sa)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n    self._internal_connect(address)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n    raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n    super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n  File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n    self.connect()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n    conn = self._new_conn()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n    retries = retries.increment(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n    raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n    get_preindex_plugin('userreports').sync_design_docs()\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n    for design in self.get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n    for design in self._get_designs():\n  File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n    db = Database(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n    self.cloudant_database.create()\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n    if not throw_on_exists and self.exists():\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n    resp = self.r_session.head(self.database_url)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n    return self.request(\"HEAD\", url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n    return super(BasicSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n    resp = real_request(self, method, url, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n    resp = super(ClientSession, self).request(\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n2025-03-24T11:03:39Z <Greenlet at 0x7ff59d19fe00: migrate_db('default')> failed with ConnectionError\n\n",
    "path": "/home/cchq/www/monolith/current/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin",
    "syspath": [
        "/tmp/ansible_django_manage_payload_wkve_1lb/ansible_django_manage_payload.zip",
        "/usr/lib/python310.zip",
        "/usr/lib/python3.10",
        "/usr/lib/python3.10/lib-dynload",
        "/usr/local/lib/python3.10/dist-packages",
        "/usr/lib/python3/dist-packages"
    ]
}```

another run check_services

``` (cchq) abello@control1:~/environments$ commcare-cloud coronams django-manage check_services
ssh abello@172.16.2.72 -t -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.5 LTS
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.
FAILURE (Took   0.11s) kafka          : No Kafka topics found
EXCEPTION (Took   0.00s) redis          : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
FAILURE (Took   0.03s) postgres       : default:commcarehq:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK No users found in postgres
EXCEPTION (Took   0.00s) couch          : Service check errored with exception 'ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__apps (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4378278be0>: Failed to establish a new connection: [Errno 111] Connection refused'))"))'
EXCEPTION (Took   0.00s) celery         : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
SUCCESS (Took   0.04s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.01s) blobdb         : Successfully saved a file to the blobdb
FAILURE (Took   5.13s) formplayer     : Could not establish a connection in time https://commcarehq.coronams.com/formplayer/serverup
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK
Connection to 172.16.2.72 closed.```

I have ran the check_services and this is the output, i got.

ssh abello@172.16.2.72 -t -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.5 LTS
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.
2025-03-24 16:02:25,969 ERROR [notify] Notify Exception: Error while checking elasticsearch cluster health
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 135, in perform_request
    response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/util/retry.py", line 527, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f3b7e504220>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/hqadmin/service_checks.py", line 128, in check_elasticsearch
    cluster_health = check_es_cluster_health()
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/hqadmin/escheck.py", line 14, in check_es_cluster_health
    return manager.cluster_health()["status"]
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/es/client.py", line 122, in cluster_health
    return self._es.cluster.health(index)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/elasticsearch5/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/elasticsearch5/client/cluster.py", line 32, in health
    return self.transport.perform_request('GET', _make_path('_cluster',
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/elasticsearch5/transport.py", line 312, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/cchq/www/monolith/releases/2025-03-22_09.18/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 144, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch5.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f3b7e504220>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f3b7e504220>: Failed to establish a new connection: [Errno 111] Connection refused)
FAILURE (Took   0.15s) kafka          : Could not connect to Kafka: NoBrokersAvailable
EXCEPTION (Took   0.00s) redis          : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
EXCEPTION (Took 241.27s) postgres       : Service check errored with exception 'OperationalError('connection to server at "172.16.2.74", port 6432 failed: FATAL:  client_login_timeout (server down)\n')'
EXCEPTION (Took   0.07s) couch          : Service check errored with exception 'ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__apps (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3b7ce0b6a0>: Failed to establish a new connection: [Errno 111] Connection refused'))"))'
EXCEPTION (Took   0.01s) celery         : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
FAILURE (Took   0.06s) elasticsearch  : Something went wrong checking cluster health
EXCEPTION (Took  60.18s) blobdb         : Service check errored with exception 'OperationalError('connection to server at "172.16.2.74", port 6432 failed: FATAL:  client_login_timeout (server down)\n')'
FAILURE (Took   5.14s) formplayer     : Could not establish a connection in time https://commcarehq.coronams.com/formplayer/serverup
SUCCESS (Took   0.01s) rabbitmq       : RabbitMQ OK
Connection to 172.16.2.72 closed. ```

Hi @alhamdu1

It appears you have networking issues with redis and couchdb.
And the installation is failing to run migrations because couchdb isn't accessible.

Is this a monolith setup? or has multiple machines?
If its multiple machines, is networking available between them?

Hey @Manish_Kangia,

I am doing a multiple machine deployment, yes there is networking available across all the hosts.

PING control1 (127.0.1.1) 56(84) bytes of data.
64 bytes from control1 (127.0.1.1): icmp_seq=1 ttl=64 time=0.027 ms

--- control1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.027/0.027/0.027/0.000 ms
PING proxy1 (172.16.2.71) 56(84) bytes of data.
64 bytes from proxy1 (172.16.2.71): icmp_seq=1 ttl=64 time=0.349 ms

--- proxy1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.349/0.349/0.349/0.000 ms
PING webworker1 (172.16.2.72) 56(84) bytes of data.
64 bytes from webworker1 (172.16.2.72): icmp_seq=1 ttl=64 time=0.260 ms

--- webworker1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.260/0.260/0.260/0.000 ms
PING webworker2 (172.16.2.73) 56(84) bytes of data.
64 bytes from webworker2 (172.16.2.73): icmp_seq=1 ttl=64 time=0.292 ms

--- webworker2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.292/0.292/0.292/0.000 ms
PING db1 (172.16.2.74) 56(84) bytes of data.
64 bytes from db1 (172.16.2.74): icmp_seq=1 ttl=64 time=0.256 ms

--- db1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.256/0.256/0.256/0.000 ms
PING db2 (172.16.2.75) 56(84) bytes of data.
64 bytes from db2 (172.16.2.75): icmp_seq=1 ttl=64 time=0.208 ms

--- db2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.208/0.208/0.208/0.000 ms

Do you know the specific ansible tags to run only the couchdb?

Just to note the install does not fail when running the redis and couchdb tasks, so i think it must be something subtle.

Hi,

I believe its this command that is failing,

Can you confirm that machine that is running this command, looks like 172.16.2.72 can ping the machine that has couch db installed on port 6379?

Though its not recommended, In order to install just couchdb, you could run
commcare-cloud <env> ap deploy_couchdb2.yml -e 'CCHQ_IS_FRESH_INSTALL=1' --skip-check

Also, can you share why you have used --skip-tags=users in the command?

Hey i ran successfully without any issues

commcare-cloud coronams ap deploy_couchdb2.yml -e 'CCHQ_IS_FRESH_INSTALL=1' --skip-check -vvv

PLAY RECAP *********************************************************************************************
172.16.2.74                : ok=54   changed=5    unreachable=0    failed=0    skipped=69   rescued=0    ignored=0
172.16.2.75                : ok=41   changed=5    unreachable=0    failed=0    skipped=62   rescued=0    ignored=0

On the --skip-tags=users my keys kept getting taken out, so logging back in was an issue.

I am now re-running the commcare-cloud coronams deploy-stack -e 'CCHQ_IS_FRESH_INSTALL=1' --skip-check --quiet -vvv

I opened and issue on this yesterday. https://github.com/dimagi/commcare-cloud/issues/6500

task path: /home/abello/commcare-cloud/src/commcare_cloud/ansible/roles/elasticsearch/tasks/main.yml:181

fatal: [172.16.2.74]: FAILED! => {
    "changed": false,
    "elapsed": 300,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 20,
            "exclude_hosts": null,
            "host": "172.16.2.74",
            "msg": null,
            "path": null,
            "port": 9200,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 300
        }
    },
    "msg": "Timeout when waiting for 172.16.2.74:9200"
}

fatal: [172.16.2.75]: FAILED! => {
    "changed": false,
    "elapsed": 300,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 20,
            "exclude_hosts": null,
            "host": "172.16.2.74",
            "msg": null,
            "path": null,
            "port": 9200,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 300
        }
    },
    "msg": "Timeout when waiting for 172.16.2.74:9200"
}

PLAY RECAP *********************************************************************************************
172.16.2.70                : ok=72   changed=5    unreachable=0    failed=0    skipped=68   rescued=0    ignored=0
172.16.2.71                : ok=73   changed=4    unreachable=0    failed=0    skipped=70   rescued=0    ignored=0
172.16.2.72                : ok=84   changed=6    unreachable=0    failed=0    skipped=83   rescued=0    ignored=0
172.16.2.73                : ok=82   changed=6    unreachable=0    failed=0    skipped=82   rescued=0    ignored=0
172.16.2.74                : ok=269  changed=16   unreachable=0    failed=1    skipped=262  rescued=0    ignored=0
172.16.2.75                : ok=250  changed=16   unreachable=0    failed=1    skipped=252  rescued=0    ignored=0

✗ Apply failed with status code 2

Hi @alhamdu1

Great! Is couch reachable now? Did the migrations run successfully?

You should add your user to the environment users so it gets added by commcare-cloud during the setup.

For elasticsearch,
In the issue you have submitted, I see

Failed to connect to the host via ssh

It seems 172.16.2.75 isn't able to connect to 172.16.2.74 on port 9200?