I'm running a deploy on an older installation (August 2021) but I'm running into an issue.
I did an update-code first for CC Cloud then looked through the changelogs and wanted to apply this update: https://dimagi.github.io/commcare-cloud/changelog/0045-upgrade-to-python-39.html
Before I can apply that, I need to make sure I'm on commit 8de1f15447be119538f2bbbcd355c3b421e33e0e so deployed that with:
cchq monolith deploy --commcare-rev 8de1f15447be119538f2bbbcd355c3b421e33e0e
I get an error during deployment with the following output:
[x.x.x.x] Executing task 'update'
[x.x.x.x] sudo: git submodule foreach "git fetch origin"
[x.x.x.x] out: Entering 'corehq/apps/hqmedia/static/hqmedia/MediaUploader'
[x.x.x.x] out: fatal: unable to connect to github.com:
[x.x.x.x] out: github.com[0: 140.82.121.3]: errno=Connection timed out
[x.x.x.x] out:
[x.x.x.x] out: fatal: run_command returned non-zero status for corehq/apps/hqmedia/static/hqmedia/MediaUploader
[x.x.x.x] out: .
[x.x.x.x] out:
Fatal error: sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""
Aborting.
!!! Parallel execution exception under host 'x.x.x.x':
Process x.x.x.x:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 240, in inner
submit(task.run(*args, **kwargs))
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 180, in inner
return func(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 45, in update
_update_code_from_previous_release(code_repo, subdir, git_env)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 72, in _update_code_from_previous_release
sudo('git submodule foreach "git fetch origin"')
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""
Fatal error: One or more hosts failed while executing task 'update'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""
Aborting.
[x.x.x.x] Executing task 'send_email'
[x.x.x.x] sudo: /home/cchq/www/monolith/releases/2022-06-17_12.47/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to monolith failed. Try resuming with fab monolith deploy:resume=yes.' 'Traceback:
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'update'"'"'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p '"'"'sudo password:'"'"' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""'
[x.x.x.x] out: fetch: -c: line 29: unexpected EOF while looking for matching `''
[x.x.x.x] out: fetch: -c: line 30: syntax error: unexpected end of file
[x.x.x.x] out:
Fatal error: sudo() received nonzero return code 1 while executing!
Requested: /home/cchq/www/monolith/releases/2022-06-17_12.47/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to monolith failed. Try resuming with fab monolith deploy:resume=yes.' 'Traceback:
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'update'"'"'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p '"'"'sudo password:'"'"' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-06-17_12.47 >/dev/null && /home/cchq/www/monolith/releases/2022-06-17_12.47/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to monolith failed. Try resuming with fab monolith deploy:resume=yes.' 'Traceback:
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py\", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 385, in execute
multiprocessing
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 275, in _execute
return task.run(*args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 172, in run
return self.wrapped(*args, **kwargs)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py\", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 417, in execute
error(err, exception=d['\"'\"'results'\"'\"'])
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py\", line 359, in error
return func(message)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py\", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '\"'\"'update'\"'\"'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach \"git fetch origin\"
Executed: sudo -S -p '\"'\"'sudo password:'\"'\"' -H -u \"cchq\" /bin/bash -l -c \"cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \\"git fetch origin\\"\"'"
Aborting.
Traceback (most recent call last):
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['results'])
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task 'update'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
*args, **kwargs
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 542, in deploy_commcare
_deploy_without_asking(skip_record)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 412, in _deploy_without_asking
traceback_string()
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 242, in send_email
f'{virtualenv_dir}/bin/python manage.py '
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!
Requested: /home/cchq/www/monolith/releases/2022-06-17_12.47/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to monolith failed. Try resuming with fab monolith deploy:resume=yes.' 'Traceback:
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
error(err, exception=d['"'"'results'"'"'])
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '"'"'update'"'"'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach "git fetch origin"
Executed: sudo -S -p '"'"'sudo password:'"'"' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \"git fetch origin\""'
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-06-17_12.47 >/dev/null && /home/cchq/www/monolith/releases/2022-06-17_12.47/python_env/bin/python manage.py send_email --to-admins --subject 'Deploy to monolith failed. Try resuming with fab monolith deploy:resume=yes.' 'Traceback:
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 398, in _deploy_without_asking
deploy_checkpoint(index, command.__name__, execute_with_timing, command)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 391, in deploy_checkpoint
fn(*args, **kwargs)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py\", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 385, in execute
multiprocessing
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 275, in _execute
return task.run(*args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 172, in run
return self.wrapped(*args, **kwargs)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py\", line 322, in _setup_release
execute_with_timing(update_code, env.deploy_ref)
File \"/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py\", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py\", line 417, in execute
error(err, exception=d['\"'\"'results'\"'\"'])
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py\", line 359, in error
return func(message)
File \"/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py\", line 55, in abort
raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task '\"'\"'update'\"'\"'
Underlying exception:
sudo() received nonzero return code 128 while executing!
Requested: git submodule foreach \"git fetch origin\"
Executed: sudo -S -p '\"'\"'sudo password:'\"'\"' -H -u \"cchq\" /bin/bash -l -c \"cd /home/cchq/www/monolith/current >/dev/null && git submodule foreach \\"git fetch origin\\"\"'"
Disconnecting from x.x.x.x... done.
>> Sending email: CommCare HQ deploy to monolith failed
Ubuntu 18.04.5 LTS
bash: python_env/bin/python: No such file or directory
Connection to x.x.x.x closed.
Skipping formplayer because commcare failed
Thanks in advance!