Hi all,
The following error is appearing while code deployment.
There was no any issue after re-indexing and upgrading to elastic search 5.6.16 before March.
TASK [deploy_hq : Ensure checkpoints safe] ***********************************************************************************************************************
fatal: [172.19.3.38]: FAILED! => {"changed": true, "cmd": ["./manage.py", "validate_kafka_pillow_checkpoints", "--print-only"], "delta": "0:00:02.799321", "end": "2024-03-11 09:16:36.929365", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2024-03-11 09:16:34.130044", "stderr": "Traceback (most recent call last):\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/./manage.py", line 190, in \n main()\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/./manage.py", line 48, in main\n execute_from_command_line(sys.argv)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line\n utility.execute()\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 395, in execute\n django.setup()\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/init.py", line 24, in setup\n apps.populate(settings.INSTALLED_APPS)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate\n app_config = AppConfig.create(entry)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/apps/config.py", line 212, in create\n mod = import_module(mod_path)\n File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "", line 1030, in _gcd_import\n File "", line 1007, in _find_and_load\n File "", line 972, in _find_and_load_unlocked\n File "", line 228, in _call_with_frames_removed\n File "", line 1030, in _gcd_import\n File "", line 1007, in _find_and_load\n File "", line 986, in _find_and_load_unlocked\n File "", line 680, in _load_unlocked\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module\n self.loader.exec_module(module)\n File "", line 850, in exec_module\n File "", line 228, in _call_with_frames_removed\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/init.py", line 1, in \n from . import ( # noqa: F401\n File "", line 1007, in _find_and_load\n File "", line 986, in _find_and_load_unlocked\n File "", line 680, in _load_unlocked\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module\n self.loader.exec_module(module)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/aggregations.py", line 42, in \n from corehq.apps.es.const import SIZE_LIMIT\n File "", line 1007, in _find_and_load\n File "", line 986, in _find_and_load_unlocked\n File "", line 680, in _load_unlocked\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module\n self.loader.exec_module(module)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/const.py", line 1, in \n from corehq.apps.es.utils import get_es_reindex_setting_value\n File "", line 1007, in _find_and_load\n File "", line 986, in _find_and_load_unlocked\n File "", line 680, in _load_unlocked\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module\n self.loader.exec_module(module)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/utils.py", line 10, in \n from corehq.util.es.elasticsearch import SerializationError\n File "", line 1007, in _find_and_load\n File "", line 986, in _find_and_load_unlocked\n File "", line 680, in _load_unlocked\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module\n self.loader.exec_module(module)\n File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/util/es/elasticsearch.py", line 9, in \n raise RuntimeError(\nRuntimeError: Elasticsearch version 2 is no longer supported. Please upgrade Elasticsearch. Details: \nReindex steps - \n\t https://commcare-cloud.readthedocs.io/en/latest/changelog/0075-reindex-all-indexes-for-es-upgrade.htmlUpgrade steps - \n\t https://commcare-cloud.readthedocs.io/en/latest/changelog/0076-upgrade-to-es-5.html\nSentry is attempting to send 2 pending error messages\nWaiting up to 2 seconds\nPress Ctrl-C to quit", "stderr_lines": ["Traceback (most recent call last):", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/./manage.py", line 190, in ", " main()", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/./manage.py", line 48, in main", " execute_from_command_line(sys.argv)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line", " utility.execute()", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 395, in execute", " django.setup()", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/init.py", line 24, in setup", " apps.populate(settings.INSTALLED_APPS)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate", " app_config = AppConfig.create(entry)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/django/apps/config.py", line 212, in create", " mod = import_module(mod_path)", " File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " File "", line 1030, in _gcd_import", " File "", line 1007, in _find_and_load", " File "", line 972, in _find_and_load_unlocked", " File "", line 228, in _call_with_frames_removed", " File "", line 1030, in _gcd_import", " File "", line 1007, in _find_and_load", " File "", line 986, in _find_and_load_unlocked", " File "", line 680, in _load_unlocked", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module", " self.loader.exec_module(module)", " File "", line 850, in exec_module", " File "", line 228, in _call_with_frames_removed", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/init.py", line 1, in ", " from . import ( # noqa: F401", " File "", line 1007, in _find_and_load", " File "", line 986, in _find_and_load_unlocked", " File "", line 680, in _load_unlocked", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module", " self.loader.exec_module(module)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/aggregations.py", line 42, in ", " from corehq.apps.es.const import SIZE_LIMIT", " File "", line 1007, in _find_and_load", " File "", line 986, in _find_and_load_unlocked", " File "", line 680, in _load_unlocked", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module", " self.loader.exec_module(module)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/const.py", line 1, in ", " from corehq.apps.es.utils import get_es_reindex_setting_value", " File "", line 1007, in _find_and_load", " File "", line 986, in _find_and_load_unlocked", " File "", line 680, in _load_unlocked", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module", " self.loader.exec_module(module)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/apps/es/utils.py", line 10, in ", " from corehq.util.es.elasticsearch import SerializationError", " File "", line 1007, in _find_and_load", " File "", line 986, in _find_and_load_unlocked", " File "", line 680, in _load_unlocked", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/python_env/lib/python3.9/site-packages/ddtrace/internal/module.py", line 216, in _exec_module", " self.loader.exec_module(module)", " File "/home/cchq/www/echis/releases/2024-03-11_09.13/corehq/util/es/elasticsearch.py", line 9, in ", " raise RuntimeError(", "RuntimeError: Elasticsearch version 2 is no longer supported. Please upgrade Elasticsearch. Details: ", "Reindex steps - ", "\t https://commcare-cloud.readthedocs.io/en/latest/changelog/0075-reindex-all-indexes-for-es-upgrade.htmlUpgrade steps - ", "\t 76. Elasticsearch upgrade from 2.4.6 to 5.6.16 — CommCareHQ Deployment documentation", "Sentry is attempting to send 2 pending error messages", "Waiting up to 2 seconds", "Press Ctrl-C to quit"], "stdout": "", "stdout_lines": }
Current elastic search version:
curl -XGET 'http://172.19.3.39:9200'
{
"name" : "es3",
"cluster_name" : "echis-es",
"cluster_uuid" : "Fff_0EABQD-fF90B6XFaMA",
"version" : {
"number" : "5.6.16",
"build_hash" : "3a740d1",
"build_date" : "2019-03-13T15:33:36.565Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
Current status:
curl -X GET '172.19.3.39:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open groups-20230524 S48N3SpEQiGu7JOhGKN31w 5 1 12277 96 840.8kb 420.4kb
red open sms-20230524 fcvzgE4-RHeZUi2CJXOF3w 5 1 4527 17 4.5mb 2.2mb
red open case-search-20230524 X-AoZZAhRbmEJ2bNNNGQsw 5 1 669368778 246361131 264.9gb 132.4gb
red open apps-20230524 xpsHrIsvSaSqHqkSQq8bkA 5 1 29228 4331 250.1mb 125mb
red open domains-20230524 kJzPIDU6RQWyu4E4VuldYw 5 1 11 3 320.4kb 160.2kb
green open users-20230524 TjGwaRjAR0y3X2nb0cq3bg 2 1 498696 210709 472.9mb 236.5mb
red open forms-20230524 qZGFMh9uQG6LzOQsv2fCGQ 5 1 3279909 545039 20.8gb 10.4gb
red open cases-20230524 xpYI4rTIS0aFYgBkwgcBew 5 1 89945515 18690285 136.5gb 68.2gb
``
Thank you,