Your right, manual install will break stuff, and I get it. I have removed manual redis and ran the check_services
ssh abello@172.16.2.72 -t -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.5 LTS
System check identified some issues:
WARNINGS:
?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.
FAILURE (Took 0.10s) kafka : No Kafka topics found
EXCEPTION (Took 0.00s) redis : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
FAILURE (Took 0.02s) postgres : default:commcarehq:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK No users found in postgres
EXCEPTION (Took 0.00s) couch : Service check errored with exception 'ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__apps (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d11928a00>: Failed to establish a new connection: [Errno 111] Connection refused'))"))'
EXCEPTION (Took 0.00s) celery : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
SUCCESS (Took 0.05s) elasticsearch : Successfully sent a doc to ES and read it back
SUCCESS (Took 0.01s) blobdb : Successfully saved a file to the blobdb
FAILURE (Took 5.17s) formplayer : Could not establish a connection in time https://commcarehq.coronams.com/formplayer/serverup
SUCCESS (Took 0.00s) rabbitmq : RabbitMQ OK
Connection to 172.16.2.72 closed.
Also re-ran the install back again
fatal: [172.16.2.72]: FAILED! => {
"changed": false,
"cmd": [
"./manage.py",
"migrate_multi",
"--noinput"
],
"invocation": {
"module_args": {
"ack_venv_creation_deprecation": null,
"app_path": "/home/cchq/www/monolith/current",
"apps": null,
"cache_table": null,
"clear": false,
"command": "migrate_multi --noinput",
"database": null,
"failfast": false,
"fixtures": null,
"link": null,
"merge": null,
"project_path": "/home/cchq/www/monolith/current",
"pythonpath": null,
"settings": null,
"skip": null,
"testrunner": null,
"virtualenv": "/home/cchq/www/monolith/current/python_env"
}
},
"msg": "stdout: \nThe following databases will be migrated:\n * default\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7ff59d19fe00: migrate_db('default')>\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n conn = connection.create_connection(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n raise err\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n sock.connect(sa)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n self._internal_connect(address)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n conn = self._new_conn()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n retries = retries.increment(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n job.get()\n File \"src/gevent/greenlet.py\", line 797, in gevent._gevent_cgreenlet.Greenlet.get\n File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 50, in reraise\n raise value.with_traceback(tb)\n File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n call_command('migrate', *args, **call_options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n return command.execute(*args, **defaults)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n output = self.handle(*args, **options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n result = super().handle(*args, **options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n post_migrate_state = executor.migrate(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n state = self._migrate_all_forwards(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n state = self.apply_migration(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n state = migration.apply(state, schema_editor)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n operation.database_forwards(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n self.code(from_state.apps, schema_editor)\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n get_preindex_plugin('userreports').sync_design_docs()\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n for design in self.get_designs():\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n for design in self._get_designs():\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n db = Database(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n self.cloudant_database.create()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n if not throw_on_exists and self.exists():\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n resp = self.r_session.head(self.database_url)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n return self.request(\"HEAD\", url, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n return super(BasicSession, self).request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n resp = real_request(self, method, url, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n resp = super(ClientSession, self).request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\n\n:stderr: System check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nSystem check identified some issues:\n\nWARNINGS:\n?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n conn = connection.create_connection(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n raise err\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n sock.connect(sa)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 586, in connect\n self._internal_connect(address)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/gevent/_socketcommon.py\", line 630, in _internal_connect\n raise _SocketError(err, strerror(err))\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n File \"/usr/lib/python3.9/http/client.py\", line 1285, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1331, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n conn = self._new_conn()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 801, in urlopen\n retries = retries.increment(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 594, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"src/gevent/greenlet.py\", line 900, in gevent._gevent_cgreenlet.Greenlet.run\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n call_command('migrate', *args, **call_options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n return command.execute(*args, **defaults)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n output = self.handle(*args, **options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n result = super().handle(*args, **options)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n post_migrate_state = executor.migrate(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n state = self._migrate_all_forwards(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n state = self.apply_migration(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n state = migration.apply(state, schema_editor)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n operation.database_forwards(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 193, in database_forwards\n self.code(from_state.apps, schema_editor)\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/apps/userreports/migrations/0001_initial.py\", line 14, in set_default_engine_ids\n get_preindex_plugin('userreports').sync_design_docs()\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 45, in sync_design_docs\n for design in self.get_designs():\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/preindex_plugins.py\", line 26, in get_designs\n for design in self._get_designs():\n File \"/home/cchq/www/monolith/releases/2025-03-22_09.18/corehq/preindex/default_plugin.py\", line 20, in _get_designs\n db = Database(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/client.py\", line 286, in __init__\n self.cloudant_database.create()\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 421, in create\n if not throw_on_exists and self.exists():\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/database.py\", line 130, in exists\n resp = self.r_session.head(self.database_url)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 624, in head\n return self.request(\"HEAD\", url, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 130, in request\n return super(BasicSession, self).request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/couchdbkit/logging.py\", line 59, in request\n resp = real_request(self, method, url, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/cloudant/_client_session.py\", line 65, in request\n resp = super(ClientSession, self).request(\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/home/cchq/www/monolith/current/python_env/lib/python3.9/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__meta (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff59b8596a0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n2025-03-24T11:03:39Z <Greenlet at 0x7ff59d19fe00: migrate_db('default')> failed with ConnectionError\n\n",
"path": "/home/cchq/www/monolith/current/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin",
"syspath": [
"/tmp/ansible_django_manage_payload_wkve_1lb/ansible_django_manage_payload.zip",
"/usr/lib/python310.zip",
"/usr/lib/python3.10",
"/usr/lib/python3.10/lib-dynload",
"/usr/local/lib/python3.10/dist-packages",
"/usr/lib/python3/dist-packages"
]
}```
another run check_services
``` (cchq) abello@control1:~/environments$ commcare-cloud coronams django-manage check_services
ssh abello@172.16.2.72 -t -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 22.04.5 LTS
System check identified some issues:
WARNINGS:
?: (staticfiles.W004) The directory '/home/cchq/www/monolith/releases/2025-03-22_09.18/node_modules' in the STATICFILES_DIRS setting does not exist.
FAILURE (Took 0.11s) kafka : No Kafka topics found
EXCEPTION (Took 0.00s) redis : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
FAILURE (Took 0.03s) postgres : default:commcarehq:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK No users found in postgres
EXCEPTION (Took 0.00s) couch : Service check errored with exception 'ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=35984): Max retries exceeded with url: /commcarehq__apps (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4378278be0>: Failed to establish a new connection: [Errno 111] Connection refused'))"))'
EXCEPTION (Took 0.00s) celery : Service check errored with exception 'ConnectionError('Error 111 connecting to localhost:6379. Connection refused.')'
SUCCESS (Took 0.04s) elasticsearch : Successfully sent a doc to ES and read it back
SUCCESS (Took 0.01s) blobdb : Successfully saved a file to the blobdb
FAILURE (Took 5.13s) formplayer : Could not establish a connection in time https://commcarehq.coronams.com/formplayer/serverup
SUCCESS (Took 0.00s) rabbitmq : RabbitMQ OK
Connection to 172.16.2.72 closed.```