Hi,
The following error appears while trying to deploy the code.
TASK [deploy_hq : Migrate databases] *************************************************************************************************************
failed: [172.19.3.38] (item=migrate_multi --noinput) => {"ansible_loop_var": "item", "changed": true, "cmd": ["./manage.py", "migrate_multi", "--noinput"], "delta": "0:00:29.911379", "end": "2024-05-17 05:52:46.520146", "item": "migrate_multi --noinput", "msg": "non-zero return code", "rc": 1, "start": "2024-05-17 05:52:16.608767", "stderr": "2024-05-17 05:52:44,210 INFO [corehq.apps.es.migration_operations] Deleting Elasticsearch index: apps-2024-03-09\nTraceback (most recent call last):\n File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 43, in migrate_db\n call_command('migrate', *args, **call_options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 194, in call_command\n return command.execute(*args, **defaults)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute\n output = self.handle(*args, **options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate.py", line 17, in handle\n result = super().handle(*args, **options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle\n post_migrate_state = executor.migrate(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate\n state = self._migrate_all_forwards(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards\n state = self.apply_migration(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration\n state = migration.apply(state, schema_editor)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply\n operation.database_forwards(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards\n self.code(from_state.apps, schema_editor)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/migration_operations.py", line 183, in run\n manager.index_delete(self.name)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/client.py", line 226, in index_delete\n self._es.indices.delete(index)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/utils.py", line 73, in _wrapped\n return func(*args, params=params, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/indices.py", line 199, in delete\n return self.transport.perform_request('DELETE', _make_path(index),\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/transport.py", line 312, in perform_request\n status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 149, in perform_request\n self._raise_error(response.status, raw_data)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/base.py", line 125, in _raise_error\n raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\nelasticsearch5.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')\n2024-05-17T05:52:44Z <Greenlet at 0x7f239b256cc0: migrate_db('default')> failed with NotFoundError", "stderr_lines": ["2024-05-17 05:52:44,210 INFO [corehq.apps.es.migration_operations] Deleting Elasticsearch index: apps-2024-03-09", "Traceback (most recent call last):", " File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 43, in migrate_db", " call_command('migrate', *args, **call_options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 194, in call_command", " return command.execute(*args, **defaults)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute", " output = self.handle(*args, **options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate.py", line 17, in handle", " result = super().handle(*args, **options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle", " post_migrate_state = executor.migrate(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate", " state = self._migrate_all_forwards(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply", " operation.database_forwards(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards", " self.code(from_state.apps, schema_editor)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/migration_operations.py", line 183, in run", " manager.index_delete(self.name)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/client.py", line 226, in index_delete", " self._es.indices.delete(index)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/utils.py", line 73, in _wrapped", " return func(*args, params=params, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/indices.py", line 199, in delete", " return self.transport.perform_request('DELETE', _make_path(index),", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/transport.py", line 312, in perform_request", " status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 149, in perform_request", " self._raise_error(response.status, raw_data)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/base.py", line 125, in _raise_error", " raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)", "elasticsearch5.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')", "2024-05-17T05:52:44Z <Greenlet at 0x7f239b256cc0: migrate_db('default')> failed with NotFoundError"], "stdout": "\nThe following databases will be migrated:\n * default\n * auditcare\n * p1\n * p2\n * p3\n * p4\n * p5\n * p6\n * p7\n * p8\n * proxy\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x7f239b256cc0: migrate_db('default')>\nTraceback (most recent call last):\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 66, in handle\n job.get()\n File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get\n File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/gevent/_compat.py", line 49, in reraise\n raise value.with_traceback(tb)\n File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 43, in migrate_db\n call_command('migrate', *args, **call_options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 194, in call_command\n return command.execute(*args, **defaults)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute\n output = self.handle(*args, **options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate.py", line 17, in handle\n result = super().handle(*args, **options)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper\n res = handle_func(*args, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle\n post_migrate_state = executor.migrate(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate\n state = self._migrate_all_forwards(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards\n state = self.apply_migration(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration\n state = migration.apply(state, schema_editor)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply\n operation.database_forwards(\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards\n self.code(from_state.apps, schema_editor)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/migration_operations.py", line 183, in run\n manager.index_delete(self.name)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/client.py", line 226, in index_delete\n self._es.indices.delete(index)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/utils.py", line 73, in _wrapped\n return func(*args, params=params, **kwargs)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/indices.py", line 199, in delete\n return self.transport.perform_request('DELETE', _make_path(index),\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/transport.py", line 312, in perform_request\n status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 149, in perform_request\n self._raise_error(response.status, raw_data)\n File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/base.py", line 125, in _raise_error\n raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\nelasticsearch5.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')", "stdout_lines": ["", "The following databases will be migrated:", " * default", " * auditcare", " * p1", " * p2", " * p3", " * p4", " * p5", " * p6", " * p7", " * p8", " * proxy", " * synclogs", "", "", "The following databases will be skipped:", " * ucr", "", "", "======================= Error During Migration =======================", "<Greenlet at 0x7f239b256cc0: migrate_db('default')>", "Traceback (most recent call last):", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 66, in handle", " job.get()", " File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get", " File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/gevent/_compat.py", line 49, in reraise", " raise value.with_traceback(tb)", " File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate_multi.py", line 43, in migrate_db", " call_command('migrate', *args, **call_options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 194, in call_command", " return command.execute(*args, **defaults)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute", " output = self.handle(*args, **options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/sql_db/management/commands/migrate.py", line 17, in handle", " result = super().handle(*args, **options)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle", " post_migrate_state = executor.migrate(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate", " state = self._migrate_all_forwards(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply", " operation.database_forwards(", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards", " self.code(from_state.apps, schema_editor)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/migration_operations.py", line 183, in run", " manager.index_delete(self.name)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/corehq/apps/es/client.py", line 226, in index_delete", " self._es.indices.delete(index)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/utils.py", line 73, in _wrapped", " return func(*args, params=params, **kwargs)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/client/indices.py", line 199, in delete", " return self.transport.perform_request('DELETE', _make_path(index),", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/transport.py", line 312, in perform_request", " status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/http_urllib3.py", line 149, in perform_request", " self._raise_error(response.status, raw_data)", " File "/home/cchq/www/echis/releases/2024-05-17_05.38/python_env/lib/python3.9/site-packages/elasticsearch5/connection/base.py", line 125, in _raise_error", " raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)", "elasticsearch5.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')"]}
skipping: [172.19.3.38] => (item=create_kafka_topics)
Elastic search Health Status:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open case-search-20230524 1EpAN3ClRKCCXfnAcr3qtQ 5 1 373553661 78847019 103.4gb 51.7gb
green open apps-20230524 Gr4XGG_mQ4iw1wNQlOq3vA 5 1 93957 66553 1.1gb 585.1mb
green open users-20230524 EqO0l5CJRouO2yv3rMUnRA 2 1 598332 348050 633.4mb 320.4mb
green open sms-20230524 G8tNLvkLSUOYHNIKvjs7Mw 5 1 35516 0 34.8mb 17.4mb
green open forms-20230524 GQ7fN8E1TpaEHSx95ryZOA 5 1 1237377 200626 10.8gb 5.3gb
green open domains-20230524 _Ya8cUzWTaSvT2_z4qocFQ 5 1 20 53 832.6kb 416.3kb
green open groups-20230524 Avx-iW7LTFKyJ5VcPDO_tA 5 1 17 2 128kb 64kb
green open cases-20230524 iQmUSkHOSD-MX7Io9JqmGA 5 1 159722487 9208610 219.1gb 109.6gb
Thank you,
