Error on deploy after code-update

I’ve done a code-update successfully followed by a commcare-cloud deploy but the deploy is producing the following error (note redacted IP):

[x.x.x.x] Executing task ‘record_successful_release’
[x.x.x.x] sudo: echo ‘/home/cchq/www/monolith/releases/2019-09-10_14.21’ >> “$(echo RELEASES.txt)”
[x.x.x.x] Executing task ‘record_successful_deploy’
[x.x.x.x] sudo: /home/cchq/www/monolith/current/python_env-3.6/bin/python manage.py record_deploy_success --user “ccc” --environment “monolith” --url https://github.com/dimagi/commcare-hq/compare/8f89d2ed8451d0ed3da3d2ea29f5b5b857c57593...4764f4b2cd35dcc10f768243391a32e6376f69d8 --minutes 16 --mail_admins
[x.x.x.x] out: /home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/psycopg2/ **init** .py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use “pip install psycopg2-binary” instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
[x.x.x.x] out: “”")
[x.x.x.x] out: 2019-09-10 14:37:24,456 INFO AXES: BEGIN LOG
[x.x.x.x] out:
[x.x.x.x] out: ---------------- Pillow Errors Reset ----------------
[x.x.x.x] out: 1 pillow errors queued for retry
[x.x.x.x] out:
[x.x.x.x] out: Traceback (most recent call last):
[x.x.x.x] out: File “manage.py”, line 195, in
[x.x.x.x] out: main()
[x.x.x.x] out: File “manage.py”, line 41, in main
[x.x.x.x] out: execute_from_command_line(sys.argv)
[x.x.x.x] out: File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/ **init** .py”, line 364, in execute_from_command_line
[x.x.x.x] out: utility.execute()
[x.x.x.x] out: File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/ **init** .py”, line 356, in execute
[x.x.x.x] out: self.fetch_command(subcommand).run_from_argv(self.argv)
[x.x.x.x] out: File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py”, line 283, in run_from_argv
[x.x.x.x] out: self.execute(*args, **cmd_options)
[x.x.x.x] out: File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py”, line 330, in execute
[x.x.x.x] out: output = self.handle(*args, **options)
[x.x.x.x] out: File “/home/cchq/www/monolith/releases/2019-09-10_14.21/corehq/apps/hqadmin/management/commands/record_deploy_success.py”, line 131, in handle
[x.x.x.x] out: if settings.SENTRY_CONFIGURED and settings.SENTRY_API_KEY:
[x.x.x.x] out: File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/conf/ **init** .py”, line 57, in  **getattr**
[x.x.x.x] out: val = getattr(self._wrapped, name)
[x.x.x.x] out: AttributeError: ‘Settings’ object has no attribute ‘SENTRY_CONFIGURED’
[x.x.x.x] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: /home/cchq/www/monolith/current/python_env-3.6/bin/python manage.py record_deploy_success --user “ccc” --environment “monolith” --url https://github.com/dimagi/commcare-hq/compare/8f89d2ed8451d0ed3da3d2ea29f5b5b857c57593...4764f4b2cd35dcc10f768243391a32e6376f69d8 --minutes 16 --mail_admins
Executed: sudo -S -p ‘sudo password:’ -H -u “cchq” /bin/bash -l -c “cd /home/cchq/www/monolith/current >/dev/null && /home/cchq/www/monolith/current/python_env-3.6/bin/python manage.py record_deploy_success --user “ccc” --environment “monolith” --url https://github.com/dimagi/commcare-hq/compare/8f89d2ed8451d0ed3da3d2ea29f5b5b857c57593...4764f4b2cd35dcc10f768243391a32e6376f69d8 --minutes 16 --mail_admins”

Aborting.
Traceback (most recent call last):
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/main.py”, line 743, in main
*args, **kwargs
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 387, in execute
multiprocessing
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 277, in _execute
return task.run(*args, **kwargs)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 174, in run
return self.wrapped(*args, **kwargs)
File “/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py”, line 704, in deploy_commcare
_deploy_without_asking(skip_record)
File “/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py”, line 552, in _deploy_without_asking
execute_with_timing(release.record_successful_deploy)
File “/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py”, line 33, in execute_with_timing
execute(fn, *args, **kwargs)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 387, in execute
multiprocessing
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 277, in _execute
return task.run(*args, **kwargs)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/tasks.py”, line 174, in run
return self.wrapped(*args, **kwargs)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/decorators.py”, line 53, in inner_decorator
return func(*args, **kwargs)
File “/home/ccc/commcare-cloud/src/commcare_cloud/fab/operations/release.py”, line 341, in record_successful_deploy
‘minutes’: str(int(delta.total_seconds() // 60))
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/network.py”, line 649, in host_prompting_wrapper
return func(*args, **kwargs)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/operations.py”, line 1109, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/operations.py”, line 946, in _run_command
error(message=msg, stdout=out, stderr=err)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/utils.py”, line 358, in error
return func(message)
File “/home/ccc/.virtualenvs/ansible/local/lib/python2.7/site-packages/fabric/utils.py”, line 54, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!

Requested: /home/cchq/www/monolith/current/python_env-3.6/bin/python manage.py record_deploy_success --user “ccc” --environment “monolith” --url https://github.com/dimagi/commcare-hq/compare/8f89d2ed8451d0ed3da3d2ea29f5b5b857c57593...4764f4b2cd35dcc10f768243391a32e6376f69d8 --minutes 16 --mail_admins
Executed: sudo -S -p ‘sudo password:’ -H -u “cchq” /bin/bash -l -c “cd /home/cchq/www/monolith/current >/dev/null && /home/cchq/www/monolith/current/python_env-3.6/bin/python manage.py record_deploy_success --user “ccc” --environment “monolith” --url https://github.com/dimagi/commcare-hq/compare/8f89d2ed8451d0ed3da3d2ea29f5b5b857c57593...4764f4b2cd35dcc10f768243391a32e6376f69d8 --minutes 16 --mail_admins”

Note that the ‘Executed: sudo -S -p ‘sudo password:’ …’ is quoted verbatim from the output - i.e. that’s not any passphrase from my vault (possibly intentional).
I assume the issue relates to
“AttributeError: ‘Settings’ object has no attribute ‘SENTRY_CONFIGURED’”

Thanks for the help!

Ed

Hi Ed, yes, that looks right - I think that was a recently introduced regression. I’ve opened a PR to fix this. In the meantime, it does look like that error was raised in the record_successful_deploy task, which occurs after all the other components of a deploy have succeeded. ie, it looks like your deploy went through, but this failure occurred while notifying you of that fact.

In the meantime, you can skip this step by running this:
$ cchq <env> deploy --skip-record
I’d expect that to succeed.

1 Like

Indeed it does - I just tested, thanks Ethan!

FYI the fix has been merged: https://github.com/dimagi/commcare-hq/pull/25332

1 Like

Thanks Simon!
Appreciate the support.