Unable to Run Commcare deploy

After updating code and pulling latest updates we are getting the following error when trying to run cchq monolith deploy:

Traceback:
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 381, in conditionally_stop_pillows_and_celery_during_migrate
execute_with_timing(db.migrate)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/operations/db.py", line 79, in migrate
sudo(f'{env.virtualenv_root}/bin/python manage.py migrate_multi --noinput')
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!

Requested: /home/cchq/www/monolith/releases/2022-10-25_11.45/python_env/bin/python manage.py migrate_multi --noinput
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-10-25_11.45 >/dev/null && /home/cchq/www/monolith/releases/2022-10-25_11.45/python_env/bin/python manage.py migrate_multi --noinput"

Prior to the error there was this output:

Running the first command generates the following release:

Your private release is located here:
/home/cchq/www/monolith/releases/2022-10-25_12.11

running the second command:

commcare-cloud monolith django-manage --release 2022-10-25_12.11 populate_lookuptablerows

Results in the following:

Detailed log output file: populate_lookuptablerows_2022-10-25T12-34-45.908148.log
Found 3819 FixtureDataItem docs and 0 LookupTableRow models
Processing Started at 2022-10-25 12:34:45
Processing [ ] 0/3819 0% -:--:-- remaining
Processing [. ] 39/3819 1% 0:00:29 remaining
Processing [............. ] 1000/3819 26% 0:00:02.055729 elapsed
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in commit
return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: insert or update on table "fixtures_lookuptablerow" violates foreign key constraint "fixtures_lookuptable_table_id_580508a8_fk_fixtures
"
DETAIL: Key (table_id)=(1892fd25-ef44-42af-aae6-703eac7acc4e) is not present in table "fixtures_lookuptable".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/manage.py", line 187, in
main()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/manage.py", line 45, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/corehq/apps/fixtures/management/commands/base.py", line 20, in handle
result = super().handle(chunk_size=chunk_size, fixup_diffs=fixup_diffs, **kw)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/corehq/apps/cleanup/management/commands/populate_sql_model_from_couch_model.py", line 339, in handle
migrate(item, logfile)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/corehq/apps/cleanup/management/commands/populate_sql_model_from_couch_model.py", line 402, in _migrate_docs
obj.save()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/transaction.py", line 246, in exit
connection.commit()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
self._commit()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
return self.connection.commit()
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/cchq/www/monolith/releases/2022-10-25_12.11/python_env/lib/python3.9/site-packages/django/db/backends/base/base.py", line 242, in commit
return self.connection.commit()
django.db.utils.IntegrityError: insert or update on table "fixtures_lookuptablerow" violates foreign key constraint "fixtures_lookuptable_table_id_580508a8_fk_fixtures
"
DETAIL: Key (table_id)=(1892fd25-ef44-42af-aae6-703eac7acc4e) is not present in table "fixtures_lookuptable".

Connection to 10.10.100.36 closed.

All changelogs to date except the latest: 2022-09-15 Update Prometheus variable have been applied to the monolith environment.

Any assistance?

Try running each of the following

commcare-cloud monolith django-manage --release 2022-10-25_12.11 populate_lookuptables
commcare-cloud monolith django-manage --release 2022-10-25_12.11 populate_lookuptablerows
commcare-cloud monolith django-manage --release 2022-10-25_12.11 populate_lookuptablerowowners

Post back here if you still encounter errors.

1 Like

Worked like a charm. Thanks Daniel!

Great! Thanks for the update.

There was a bug in the automated migration that caused your first run of populate_lookuptablerows to fail, and the commands I provided above have allowed you to work around it. The bug should be fixed (for others) by Do not use transaction for Couch to SQL migrations by millerdev · Pull Request #32283 · dimagi/commcare-hq · GitHub

1 Like