Hi,
We've been unable to run a successful monolith deploy after updating code:
The deploy initially returns the following before attempting to proceed:
at which point it fails after attempting to send an email that pre-indexing may still be running in the background with the floowing output:
[10.10.100.36] sudo: /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python /home/cchq/www/monolith/releases/2022-09-06_16.41/manage.py preindex_everything --check
[10.10.100.36] out: Preindex is not yet complete.
[10.10.100.36] out:
[10.10.100.36] out: It could either still be running (most common),
[10.10.100.36] out: or it could have died without completing (less common).
[10.10.100.36] out:[10.10.100.36] sudo: /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'
[10.10.100.36] out: Traceback (most recent call last):
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/manage.py", line 208, in
[10.10.100.36] out: main()
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/manage.py", line 45, in main
[10.10.100.36] out: execute_from_command_line(sys.argv)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
[10.10.100.36] out: utility.execute()
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
[10.10.100.36] out: self.fetch_command(subcommand).run_from_argv(self.argv)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
[10.10.100.36] out: self.execute(*args, **cmd_options)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
[10.10.100.36] out: output = self.handle(*args, **options)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/corehq/apps/hqadmin/management/commands/send_email.py", line 45, in handle
[10.10.100.36] out: mail_admins(subject, message, html_message=message if is_html else None)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/mail/init.py", line 104, in mail_admins
[10.10.100.36] out: mail.send(fail_silently=fail_silently)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/mail/message.py", line 284, in send
[10.10.100.36] out: return self.get_connection(fail_silently).send_messages([self])
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 109, in send_messages
[10.10.100.36] out: sent = self._send(message)
[10.10.100.36] out: File "/home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 125, in _send
[10.10.100.36] out: self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n'))
[10.10.100.36] out: File "/usr/lib/python3.9/smtplib.py", line 901, in sendmail
[10.10.100.36] out: raise SMTPRecipientsRefused(senderrs)
[10.10.100.36] out: smtplib.SMTPRecipientsRefused: {'commcarehq-ops+admins@example.com': (553, b'5.7.1 notifications@mail.gov.mw: Sender address rejected: not owned by user noreply@mail.gov.mw')}
[10.10.100.36] out:Fatal error: sudo() received nonzero return code 1 while executing!
Requested: /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-09-06_16.41 >/dev/null && /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'"Aborting.
Traceback (most recent call last):
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/operations/db.py", line 38, in ensure_preindex_completion
raise PreindexNotFinished()
commcare_cloud.fab.exceptions.PreindexNotFinishedDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
*args, **kwargs
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 542, in deploy_commcare
_deploy_without_asking(skip_record)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 402, in _deploy_without_asking
("Preindexing is taking a while, so hold tight "
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/nmtambalika/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 242, in send_email
f'{virtualenv_dir}/bin/python manage.py '
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/nmtambalika/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!Requested: /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-09-06_16.41 >/dev/null && /home/cchq/www/monolith/releases/2022-09-06_16.41/python_env/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'"
Disconnecting from 10.10.100.36... done.Sending email: CommCare HQ deploy to monolith failed
Ubuntu 18.04.6 LTS
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/manage.py", line 190, in
main()
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/manage.py", line 44, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/corehq/apps/hqadmin/management/commands/send_email.py", line 45, in handle
mail_admins(subject, message, html_message=message if is_html else None)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/mail/init.py", line 104, in mail_admins
mail.send(fail_silently=fail_silently)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/mail/message.py", line 284, in send
return self.get_connection(fail_silently).send_messages([self])
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 109, in send_messages
sent = self._send(message)
File "/home/cchq/www/monolith/releases/2022-06-24_12.27/python_env/lib/python3.9/site-packages/django/core/mail/backends/smtp.py", line 125, in _send
self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n'))
File "/usr/lib/python3.9/smtplib.py", line 901, in sendmail
raise SMTPRecipientsRefused(senderrs)
smtplib.SMTPRecipientsRefused: {'commcarehq-ops+admins@example.com': (553, b'5.7.1 notifications@mail.gov.mw: Sender address rejected: not owned by user noreply@mail.gov.mw')}
Connection to 10.10.100.36 closed.
Skipping formplayer because commcare failed
we're not sure where the email address commcarehq-ops+admins@example.com has been defined and if this could be what is causing the deploy to fail, our proxy.yml and vault.yml files explicitly have noreply@health.gov.mw defined as server email address and email login.