Unable to deploy prod stable b23df94 25 Apr 2024

Developers
1

I'm trying to deploy b23df94 but am receiving a migration error:

TASK [deploy_hq : Migrate databases] *****************************************************************************************************************************
failed: [10.3.0.4] (item=migrate_multi --noinput) => {"ansible_loop_var": "item", "changed": true, "cmd": ["./manage.py", "migrate_multi", "--noinput"], "delta": "0:00:25.049732", "end": "2024-04-27 17:38:40.769976", "item": "migrate_multi --noinput", "msg": "non-zero return code", "rc": 1, "start": "2024-04-27 17:38:15.720244", "stderr": "Traceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\npsycopg2.errors.UndefinedColumn: column users_invitation.tableau_role does not exist\nLINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...\n                                                             ^\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 244, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 117, in migrate\n    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 147, in _migrate_all_forwards\n    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 227, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 126, in apply\n    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 190, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/django_migrations.py\", line 121, in _inner\n    return migration_fn(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/migrations/0058_createprofileanduserdata_copylocation.py\", line 11, in copy_invitation_supply_point\n    call_command('copy_invitation_supply_point')\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/management/commands/copy_invitation_supply_point.py\", line 16, in handle\n    for invitation in with_progress_bar(\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/log.py\", line 273, in with_progress_bar\n    for i, x in enumerate(iterable, start=1):\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/queries.py\", line 114, in queryset_to_iterator\n    while docs:\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 284, in __bool__\n    self._fetch_all()\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 1324, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 51, in __iter__\n    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1175, in execute_sql\n    cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 66, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 75, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 90, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\ndjango.db.utils.ProgrammingError: column users_invitation.tableau_role does not exist\nLINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...\n                                                             ^\n\n2024-04-27T17:38:39Z <Greenlet at 0x740fa4425d60: migrate_db('default')> failed with ProgrammingError", "stderr_lines": ["Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "psycopg2.errors.UndefinedColumn: column users_invitation.tableau_role does not exist", "LINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...", "                                                             ^", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 244, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 117, in migrate", "    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 147, in _migrate_all_forwards", "    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 227, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 126, in apply", "    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 190, in database_forwards", "    self.code(from_state.apps, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/django_migrations.py\", line 121, in _inner", "    return migration_fn(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/migrations/0058_createprofileanduserdata_copylocation.py\", line 11, in copy_invitation_supply_point", "    call_command('copy_invitation_supply_point')", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/management/commands/copy_invitation_supply_point.py\", line 16, in handle", "    for invitation in with_progress_bar(", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/log.py\", line 273, in with_progress_bar", "    for i, x in enumerate(iterable, start=1):", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/queries.py\", line 114, in queryset_to_iterator", "    while docs:", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 284, in __bool__", "    self._fetch_all()", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 1324, in _fetch_all", "    self._result_cache = list(self._iterable_class(self))", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 51, in __iter__", "    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1175, in execute_sql", "    cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 66, in execute", "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 75, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 90, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "django.db.utils.ProgrammingError: column users_invitation.tableau_role does not exist", "LINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...", "                                                             ^", "", "2024-04-27T17:38:39Z <Greenlet at 0x740fa4425d60: migrate_db('default')> failed with ProgrammingError"], "stdout": "\nThe following databases will be migrated:\n * default\n * p1\n * p2\n * proxy\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\nFound 0 RepeatRecord documents to migrate.\nStarted at 2024-04-27 17:38:39\nProcessing [..................................................] 0/0 100% 0:00:00.000059 elapsed\rProcessing [..................................................] 0/0 100% 0:00:00.002099 elapsed\n\n======================= Error During Migration =======================\n<Greenlet at 0x740fa4425d60: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\npsycopg2.errors.UndefinedColumn: column users_invitation.tableau_role does not exist\nLINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...\n                                                             ^\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 805, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 49, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 244, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 117, in migrate\n    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 147, in _migrate_all_forwards\n    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 227, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 126, in apply\n    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 190, in database_forwards\n    self.code(from_state.apps, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/django_migrations.py\", line 121, in _inner\n    return migration_fn(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/migrations/0058_createprofileanduserdata_copylocation.py\", line 11, in copy_invitation_supply_point\n    call_command('copy_invitation_supply_point')\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/management/commands/copy_invitation_supply_point.py\", line 16, in handle\n    for invitation in with_progress_bar(\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/log.py\", line 273, in with_progress_bar\n    for i, x in enumerate(iterable, start=1):\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/queries.py\", line 114, in queryset_to_iterator\n    while docs:\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 284, in __bool__\n    self._fetch_all()\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 1324, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 51, in __iter__\n    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1175, in execute_sql\n    cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 66, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 75, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 90, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\ndjango.db.utils.ProgrammingError: column users_invitation.tableau_role does not exist\nLINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...\n                                                             ^", "stdout_lines": ["", "The following databases will be migrated:", " * default", " * p1", " * p2", " * proxy", " * synclogs", "", "", "The following databases will be skipped:", " * ucr", "", "Found 0 RepeatRecord documents to migrate.", "Started at 2024-04-27 17:38:39", "Processing [..................................................] 0/0 100% 0:00:00.000059 elapsed", "Processing [..................................................] 0/0 100% 0:00:00.002099 elapsed", "", "======================= Error During Migration =======================", "<Greenlet at 0x740fa4425d60: migrate_db('default')>", "Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "psycopg2.errors.UndefinedColumn: column users_invitation.tableau_role does not exist", "LINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...", "                                                             ^", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle", "    job.get()", "  File \"src/gevent/greenlet.py\", line 805, in gevent._gevent_cgreenlet.Greenlet.get", "  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 49, in reraise", "    raise value.with_traceback(tb)", "  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 89, in wrapped", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 244, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 117, in migrate", "    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 147, in _migrate_all_forwards", "    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 227, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 126, in apply", "    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/migrations/operations/special.py\", line 190, in database_forwards", "    self.code(from_state.apps, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/django_migrations.py\", line 121, in _inner", "    return migration_fn(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/migrations/0058_createprofileanduserdata_copylocation.py\", line 11, in copy_invitation_supply_point", "    call_command('copy_invitation_supply_point')", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 181, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/apps/users/management/commands/copy_invitation_supply_point.py\", line 16, in handle", "    for invitation in with_progress_bar(", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/log.py\", line 273, in with_progress_bar", "    for i, x in enumerate(iterable, start=1):", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/corehq/util/queries.py\", line 114, in queryset_to_iterator", "    while docs:", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 284, in __bool__", "    self._fetch_all()", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 1324, in _fetch_all", "    self._result_cache = list(self._iterable_class(self))", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/query.py\", line 51, in __iter__", "    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1175, in execute_sql", "    cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 66, in execute", "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 75, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 90, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/home/cchq/www/monolith/releases/2024-04-27_17.28/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "django.db.utils.ProgrammingError: column users_invitation.tableau_role does not exist", "LINE 1: ...ofile_id\", \"users_invitation\".\"custom_user_data\", \"users_inv...", "                                                             ^"]}
skipping: [10.3.0.4] => (item=create_kafka_topics)

"UndefinedColumn: column users_invitation.tableau_role does not exist"

Current version is 6ac971b (2024-03-21_09.56-production-deploy) and CC Cloud is up to date with 1266e44 - Fri Apr 26.

I assume there's a script missing? Should I be deploying an earlier release first?
Thanks!

2

Hi Ed,

Sorry for the inconvenience. We have implemented a fix. Please reference this changelog for the steps to fix this issue.

In summary, the problem is that there are migration dependency ordering issues in the user table.

1 Like
3

Thanks for the feedback @Jonahtang, appreciate it! I'll follow up when I deploy again.
Ed

4

Unfortunately no-go. I assume this may be a knock-on issue from the prior attempted deploy where some migrations went through but others didn't?

I now receive this error:

TASK [Run run_migrations] *******************************************************************************************************************************************************************

TASK [deploy_hq : Migrate databases] ********************************************************************************************************************************************************
failed: [10.3.0.4] (item=migrate_multi --noinput) => {"ansible_loop_var": "item", "changed": true, "cmd": ["./manage.py", "migrate_multi", "--noinput"], "delta": "0:00:22.133012", "end": "2024-05-04 18:34:59.959447", "item": "migrate_multi --noinput", "msg": "non-zero return code", "rc": 1, "start": "2024-05-04 18:34:37.826435", "stderr": "Traceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute\n    return self.cursor.execute(sql)\npsycopg2.errors.DuplicateColumn: column \"location_id\" of relation \"users_invitation\" already exists\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/operations/fields.py\", line 108, in database_forwards\n    schema_editor.add_field(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 713, in add_field\n    self.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/postgresql/schema.py\", line 48, in execute\n    return super().execute(sql, None)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 201, in execute\n    cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute\n    return self.cursor.execute(sql)\ndjango.db.utils.ProgrammingError: column \"location_id\" of relation \"users_invitation\" already exists\n\n2024-05-04T18:34:58Z <Greenlet at 0x733ace023220: migrate_db('default')> failed with ProgrammingError", "stderr_lines": ["Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute", "    return self.cursor.execute(sql)", "psycopg2.errors.DuplicateColumn: column \"location_id\" of relation \"users_invitation\" already exists", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/operations/fields.py\", line 108, in database_forwards", "    schema_editor.add_field(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 713, in add_field", "    self.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/postgresql/schema.py\", line 48, in execute", "    return super().execute(sql, None)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 201, in execute", "    cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute", "    return self._execute_with_wrappers(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute", "    return self.cursor.execute(sql)", "django.db.utils.ProgrammingError: column \"location_id\" of relation \"users_invitation\" already exists", "", "2024-05-04T18:34:58Z <Greenlet at 0x733ace023220: migrate_db('default')> failed with ProgrammingError"], "stdout": "\nThe following databases will be migrated:\n * default\n * p1\n * p2\n * proxy\n * synclogs\n\n\nThe following databases will be skipped:\n * ucr\n\n\n======================= Error During Migration =======================\n<Greenlet at 0x733ace023220: migrate_db('default')>\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute\n    return self.cursor.execute(sql)\npsycopg2.errors.DuplicateColumn: column \"location_id\" of relation \"users_invitation\" already exists\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle\n    job.get()\n  File \"src/gevent/greenlet.py\", line 805, in gevent._gevent_cgreenlet.Greenlet.get\n  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 49, in reraise\n    raise value.with_traceback(tb)\n  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db\n    call_command('migrate', *args, **call_options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command\n    return command.execute(*args, **defaults)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate.py\", line 17, in handle\n    result = super().handle(*args, **options)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle\n    post_migrate_state = executor.migrate(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate\n    state = self._migrate_all_forwards(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards\n    state = self.apply_migration(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration\n    state = migration.apply(state, schema_editor)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply\n    operation.database_forwards(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/operations/fields.py\", line 108, in database_forwards\n    schema_editor.add_field(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 713, in add_field\n    self.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/postgresql/schema.py\", line 48, in execute\n    return super().execute(sql, None)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 201, in execute\n    cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute\n    return self.cursor.execute(sql)\ndjango.db.utils.ProgrammingError: column \"location_id\" of relation \"users_invitation\" already exists", "stdout_lines": ["", "The following databases will be migrated:", " * default", " * p1", " * p2", " * proxy", " * synclogs", "", "", "The following databases will be skipped:", " * ucr", "", "", "======================= Error During Migration =======================", "<Greenlet at 0x733ace023220: migrate_db('default')>", "Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute", "    return self.cursor.execute(sql)", "psycopg2.errors.DuplicateColumn: column \"location_id\" of relation \"users_invitation\" already exists", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 66, in handle", "    job.get()", "  File \"src/gevent/greenlet.py\", line 805, in gevent._gevent_cgreenlet.Greenlet.get", "  File \"src/gevent/greenlet.py\", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/gevent/_compat.py\", line 49, in reraise", "    raise value.with_traceback(tb)", "  File \"src/gevent/greenlet.py\", line 908, in gevent._gevent_cgreenlet.Greenlet.run", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate_multi.py\", line 43, in migrate_db", "    call_command('migrate', *args, **call_options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/__init__.py\", line 194, in call_command", "    return command.execute(*args, **defaults)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/corehq/sql_db/management/commands/migrate.py\", line 17, in handle", "    result = super().handle(*args, **options)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", "    post_migrate_state = executor.migrate(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", "    state = self._migrate_all_forwards(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", "    state = self.apply_migration(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", "    state = migration.apply(state, schema_editor)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", "    operation.database_forwards(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/migrations/operations/fields.py\", line 108, in database_forwards", "    schema_editor.add_field(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 713, in add_field", "    self.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/postgresql/schema.py\", line 48, in execute", "    return super().execute(sql, None)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/base/schema.py\", line 201, in execute", "    cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute", "    return self._execute_with_wrappers(", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/home/cchq/www/monolith/releases/2024-05-04_18.23/python_env/lib/python3.9/site-packages/django/db/backends/utils.py\", line 87, in _execute", "    return self.cursor.execute(sql)", "django.db.utils.ProgrammingError: column \"location_id\" of relation \"users_invitation\" already exists"]}
skipping: [10.3.0.4] => (item=create_kafka_topics)

"DuplicateColumn: column "location_id" of relation "users_invitation" already exists"

For reference, I was able to install c3f318f3 as per changelog 80: Copy Supply Point data to Invitation model Location Field. Changelog 80 seemed to go through fine, but when deploying e2df5e5 I get the above failure.

I'm reluctant to delete the column as I imagine the rabbit hole may go deeper and there may be others that are being inserted that already exist. Any assistance appreciated!
Thanks

5

Sorry Ed, there was an error in the changelog directions. The "min_commcare_version" required version is " b2b03835", not "c3f318f3". This changelog has been corrected.

In short, the migration file name was changed between "c3f318f3", and "b2b03835" causing the same migration to be run twice.

To fix the state you are in:

  • Check out version " c3f318f3"
  • Revert migrations for users app to 0057 (to undo 0058)
  • Follow the updated changelog instructions
1 Like
6

Thanks for the update @Jonahtang I'll deploy this weekend and report back.
Ed

7

Hi @Jonahtang, unfortunately we're still having issues with this. In short, I was able to follow your suggestion of checking out c3f318f3:

sudo su - cchq
cd www/monolith/current
git checkout c3f318f3

...and reverting migrations for users app to 0057

source python_env-3.9/bin/activate
python manage.py migrate users 0057

...then following the changelog instructions which include:
Deploy b2b03835 first before running:

commcare-cloud django-manage --tmux copy_invitation_supply_point

The above worked fine, but I tried to upgrade to the latest commit (f7ade99) and I receive this error:

Basically, for some reason, elasticsearch didn't appear to be running at that point. I restarted ES and re-ran the deploy but now I'm getting:

In a nutshell:
"elasticsearch5.exceptions.RequestError: TransportError(400, 'index_already_exists_exception', 'index [apps-2024-03-09/Lur5-Bh_SwKpHt49Ay5Ivg] already exists')"]}"

Your help is appreciated!
Ed

8

Hey @erobinson !

I hope you're doing well! Those were temporary indices created a few months ago and then rolled back. We added them back to ensure no issues for anyone who deployed around that time, as the migration would have already been registered in Django and would not run again.

It's surprising to see the indices already exist on your system and the migration is also trying to create them.

A simple fix would be faking the migration as the indices are no longer required. Fake the migration using the command below after activating the Python virtual environment on your django-manage machine.

Steps:

sudo su - cchq
cd www/monolith/current
source python_env-3.9/bin/activate
python manage.py migrate --fake es 0009

Then, you can run the entire es app migrations:

python manage.py migrate es

These steps should unblock your deploy. Let us know if you face any other issue. We would be happy to help you.

The steps you have mentioned in the comments above should not have resulted in this situation. Do you remember deploying between 12th March and 14th March? (Its totally fine if you don't) And is there anything specific you remember doing to this env which might have resulted in altering migrations?

Thanks.

1 Like
9

Hi @aphulera , no, I wouldn't have deployed between 12th and 14th March I don't think. Our deployments tend to happen over the weekend, so for March, I deployed on 30 Mar:
6ac971bd – 21 Mar 2024 (CCHQ) and
65d9cade – 29 Mar 2024 (CC Cloud).

Another issue has come up. We have a client that submitted data yesterday but apparently it's not showing up in the app. I assume it could be related to the ES indexes and potentially this failed upgrade?

10

Running the python manage.py migrate --fake es 0009 I receive this error:

CommandError: Cannot find a migration matching '0009' from app 'es'.

Attempting a re-deploy of the same version f7ade99 results in the same error:

"elasticsearch5.exceptions.RequestError: TransportError(400, 'index_already_exists_exception', 'index [apps-2024-03-09/Lur5-Bh_SwKpHt49Ay5Ivg] already exists')

11

Ahh! Apologies @erobinson. I have provided the incomplete migration name.

This should work

python manage.py migrate --fake es  0009_add_indices_for_reindex_in_es5
12

I don't think it should be related to the change as the indices are not in use yet. They are created for an upgrade that is planned in near future. The working of Commcare should not be affected by this change in any way.

13

Hi @aphulera, I don't seem to have that migration (0009_add_indices_for_reindex_in_es5) in my current deployment (b2b03835) - this is what I have for ES:

./corehq/apps/es/migrations/0001_bootstrap_es_indexes.py
./corehq/apps/es/migrations/0002_add_tombstones.py
./corehq/apps/es/migrations/0003_add_assigned_location_ids.py
./corehq/apps/es/migrations/0004_make_new_indexes.py
./corehq/apps/es/migrations/0005_add_epoch_as_valid_date_to_forms.py
./corehq/apps/es/migrations/0006_verify_es2_indices_reindexed.py
./corehq/apps/es/migrations/0007_init_indices_for_fresh_es_5.py
./corehq/apps/es/migrations/0008_add_doc_id_to_all_mappings.py

Should I perhaps just try deleting that index (apps-2024-03-09/Lur5-Bh_SwKpHt49Ay5Ivg) if it's no longer required, then deploying again?

**EDIT - if I cat indices I see:

green open forms-20230524 MLyRSMXISPe8HWGMH8HwVw 5 0 339811 4006 1.1gb 1.1gb
green open apps-20230524 5ANxtK4QQ2uIPZAVk6rE-w 5 0 30753 2471 140.1mb 140.1mb
green open domains-20230524 mYeytKTjRpm2KdPHB5Yj-A 5 0 23 1 348.4kb 348.4kb
green open groups-20230524 o14_BR56QGGJV0bof_-XIw 5 0 81 22 121.1kb 121.1kb
green open case-search-20230524 Xr8lJ60yQPqtYXigHdXXJg 5 0 12935092 165944 1.2gb 1.2gb
green open apps-2024-03-09 Lur5-Bh_SwKpHt49Ay5Ivg 5 0 0 0 955b 955b
green open users-20230524 RFHIyZD-T0yRgpWd3s4IVQ 2 0 10892 938 3.7mb 3.7mb
green open cases-20230524 oGBxz_d0ToKeoDJT0DUhhA 5 0 1134812 19082 504.9mb 504.9mb
green open sms-20230524 HWcbmOlHSNGC_7huzEfrzg 5 0 0 0 1kb 1kb

Thanks

14

The migrations should be on the latest branch on master but looking at your index states definitely it makes much more sense to delete the index

You can delete the index by

curl -X DELETE "http://<es_host>:<es_port>/apps-2024-03-09"
1 Like
15

Thanks Amit, I'll have a go this weekend likely.
Ed

16

Hi @aphulera I managed to delete that Index OK:

(cchq) ccc@monolith:~/commcare-cloud$ curl -XGET http://10.3.0.4:9200/_cat/indices
green open forms-20230524       MLyRSMXISPe8HWGMH8HwVw 5 0   339811   4069   1.1gb   1.1gb
green open apps-20230524        5ANxtK4QQ2uIPZAVk6rE-w 5 0    30753   2471 140.1mb 140.1mb
green open domains-20230524     mYeytKTjRpm2KdPHB5Yj-A 5 0       23      1 373.3kb 373.3kb
green open groups-20230524      o14_BR56QGGJV0bof_-XIw 5 0       81     22 121.1kb 121.1kb
green open case-search-20230524 Xr8lJ60yQPqtYXigHdXXJg 5 0 12935092 166402   1.2gb   1.2gb
green open users-20230524       RFHIyZD-T0yRgpWd3s4IVQ 2 0    10892   1111   3.9mb   3.9mb
green open cases-20230524       oGBxz_d0ToKeoDJT0DUhhA 5 0  1134812  19107 504.9mb 504.9mb
green open sms-20230524         HWcbmOlHSNGC_7huzEfrzg 5 0        0      0     1kb     1kb

I then deployed again, but got the first ES failure I received (in the opening post) during migrations which is an ES timeout:

Creating Elasticsearch index: apps-2024-03-09
...
elasticsearch5.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='10.3.0.4', port=9200): Read timed out. (read timeout=30))

If I deploy again after restarting ES, (that apps-2024-03-09 index is now there), I get the index already exists error in the Migrate databases step:

elasticsearch5.exceptions.RequestError: TransportError(400, 'index_already_exists_exception', 'index [apps-2024-03-09/8hw5KKLgRs6Xb5JBu5P_og] already exists')

I assume it's recreating that same index at some point during the deploy, but if the index isn't there when I deploy, it seems that something in that process is putting the ES health into red and there's a timeout, halting the process after the first 2024-03-09 index creation.

17

What I tried next was to fake the 0009 migration by switching to the newer release that had failed and running the migrate --fake command:

cd ~/www/monolith/releases/2024-05-18_04.41
source python_env-3.9/bin/activate
python manage.py migrate --fake es  0009_add_indices_for_reindex_in_es5

That ran OK

Operations to perform:
  Target specific migration: 0009_add_indices_for_reindex_in_es5, from es
Running migrations:
  Applying es.0009_add_indices_for_reindex_in_es5... FAKED

I then deployed the latest production release:

commcare-cloud monolith deploy --commcare-rev 9ed2bd1

Now I receive an error indicating another index is missing when trying to delete it (apps-2024-03-09)

Deleting Elasticsearch index: case-search-2024-03-09
.....
elasticsearch5.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index'

A _cat/indices returns:

(cchq) ccc@monolith:~/commcare-cloud$  curl -XGET http://10.3.0.4:9200/_cat/indices
green open forms-20230524       MLyRSMXISPe8HWGMH8HwVw 5 0   339811   4121   1.1gb   1.1gb
green open apps-20230524        5ANxtK4QQ2uIPZAVk6rE-w 5 0    30753   2471 140.1mb 140.1mb
green open domains-20230524     mYeytKTjRpm2KdPHB5Yj-A 5 0       23      1 373.3kb 373.3kb
green open groups-20230524      o14_BR56QGGJV0bof_-XIw 5 0       82     22 124.4kb 124.4kb
green open case-search-20230524 Xr8lJ60yQPqtYXigHdXXJg 5 0 12935092 166402   1.2gb   1.2gb
green open users-20230524       RFHIyZD-T0yRgpWd3s4IVQ 2 0    10892   1111   3.9mb   3.9mb
green open cases-20230524       oGBxz_d0ToKeoDJT0DUhhA 5 0  1134812  19107 504.9mb 504.9mb
green open sms-20230524         HWcbmOlHSNGC_7huzEfrzg 5 0        0      0     1kb     1kb
18

So after some testing, it seems there's a consistent pattern here. If I deploy without the "2024-03-09" indices:

(cchq) ccc@monolith:~/commcare-cloud$ curl -XGET http://10.3.0.4:9200/_cat/indices
green open forms-20230524       MLyRSMXISPe8HWGMH8HwVw 5 0   339811   4069   1.1gb   1.1gb
green open apps-20230524        5ANxtK4QQ2uIPZAVk6rE-w 5 0    30753   2471 140.1mb 140.1mb
green open domains-20230524     mYeytKTjRpm2KdPHB5Yj-A 5 0       23      1 373.3kb 373.3kb
green open groups-20230524      o14_BR56QGGJV0bof_-XIw 5 0       82     22 124.4kb 124.4kb
green open case-search-20230524 Xr8lJ60yQPqtYXigHdXXJg 5 0 12935092 166402   1.2gb   1.2gb
green open users-20230524       RFHIyZD-T0yRgpWd3s4IVQ 2 0    10892   1111   3.9mb   3.9mb
green open cases-20230524       oGBxz_d0ToKeoDJT0DUhhA 5 0  1134812  19107 504.9mb 504.9mb
green open sms-20230524         HWcbmOlHSNGC_7huzEfrzg 5 0        0      0     1kb     1kb

...the deploy seems to consistently fail during the migration with an ES read timeout

Creating Elasticsearch index: apps-2024-03-09
...
elasticsearch5.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='10.3.0.4', port=9200): Read timed out. (read timeout=30))

If I check_services after the failure, I'm told

FAILURE (Took 0.00s) elasticsearch : Cluster health at red

If I restart ES and get it going again and check indices, I see a new index that was created by the previous migration I assume:

(cchq) ccc@monolith:~/commcare-cloud$ curl -XGET http://10.3.0.4:9200/_cat/indices
green open forms-20230524       MLyRSMXISPe8HWGMH8HwVw 5 0   339811   4169   1.1gb   1.1gb
green open apps-20230524        5ANxtK4QQ2uIPZAVk6rE-w 5 0    30753   2471 140.1mb 140.1mb
green open **apps-2024-03-09**      ml59PWzGT4KLIcAyNQ64Ww 5 0        0      0    810b    810b
green open domains-20230524     mYeytKTjRpm2KdPHB5Yj-A 5 0       23      1 373.3kb 373.3kb
green open groups-20230524      o14_BR56QGGJV0bof_-XIw 5 0       81     22 121.1kb 121.1kb
green open case-search-20230524 Xr8lJ60yQPqtYXigHdXXJg 5 0 12935092 166402   1.2gb   1.2gb
green open users-20230524       RFHIyZD-T0yRgpWd3s4IVQ 2 0    10892   1111   3.9mb   3.9mb
green open cases-20230524       oGBxz_d0ToKeoDJT0DUhhA 5 0  1134812  19107 504.9mb 504.9mb
green open sms-20230524         HWcbmOlHSNGC_7huzEfrzg 5 0        0      0     1kb     1kb

If I try redeploying again, I get the 'index already exists' error... if I fake the 0009 migration, I get "no such index" error during migration with "Deleting Elasticsearch index: apps-2024-03-09"

19

Hey Ed!

This would be so frustrating for you, I really feel sorry for about the situation.

I have a added a fix on last Friday in our ES migration framework which would ignore the not necessary indices and would delete them only if they exists.

Any specific reason why you want to deploy 9ed2bd1? If you are okay, would you be able to make a deploy with the latest master on commcare-hq or any commit greater than or equals to
cd6b2c03f92ea24890848fe7b01e12f8e5d85923. This would surely fix the index not exists and index already exists failures.

I am sorry again that this turned out hairy for you.

Regards,
Amit!

1 Like
20

Thanks for this Amit and no worries, sometimes stuff happens! The reason I was deploying 9ed2bd1 is because it was the latest verified production deploy as per the tags: Tags · dimagi/commcare-hq · GitHub

I've a bad experience with deploying the absolute latest, or a release not tagged as production in the past and since this is a production server, I usually hang back to the second last verified production release if the latest is only days old. I've seen a few weeks where successive production releases are released daily where I imagine issues could have been picked up after deployment, so I often pick a verified release that has been deployed for a week or so.

Ill deploy later this week or on the weekend as soon as a new verified production release appears in the tags.
Thanks!

1 Like