Requirement file missing during Commcare deploy

Hello everyone,

I encounter an error while updating commcare with the deploy command.

[x.x.x.x] sudo: virtualenv-clone /home/cchq/www/monolith/releases/2020-09-29_11.57/python_env-3.6 /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6
[x.x.x.x] sudo: sed -i -e "s~/home/cchq/www/monolith/current/python_env-3.6~/home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6~g" $(find /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/ -type f)
[x.x.x.x] sudo: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/activate && pip uninstall -r /home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt --yes
[x.x.x.x] out: ERROR: Could not open requirements file: [Errno 2] Aucun fichier ou dossier de ce type: '/home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt'
[x.x.x.x] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/activate && pip uninstall -r /home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt --yes
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-07-01_18.34 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/activate && pip uninstall -r /home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt --yes"

Aborting.
!!! Parallel execution exception under host u'x.x.x.x':

...
Underlying exception:
sudo() received nonzero return code 1 while executing!

...
Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/activate && pip uninstall -r /home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt --yes
Executed: sudo -S -p '"'"'sudo password:'"'"' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-07-01_18.34 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2022-07-01_18.34/python_env-3.6/bin/activate && pip uninstall -r /home/cchq/www/monolith/releases/2022-07-01_18.34/requirements/uninstall-requirements.txt --yes"' --environment monolith"
...

Disconnecting from x.x.x.x... done.

Someone could help me ?

Thanks in advance.

Hello,

If the other commands in the deploy command worked correctly, the requirements file should exist in that expected location. What exact series of commands did you run when you got this error?

Hi Sravan_Reddy,

I try to follow the documentation here https://commcare-cloud.readthedocs.io/en/latest/operations/2-deploys.html#step-1-update-commcare-cloud

Here are the commands I executed before I got this error :

  • update-code
  • commcare-cloud monolith deploy

Before these two commands I updated the url format (https://github.com/) of the submodules on current release.

Thanks,

Have you been able to successfully deploy in the past? If so, do the requirements files exist in the current release?

If you have not deployed before have you run the Ansible deploy?

$ commcare-cloud monolith deploy-stack --skip-check -e 'CCHQ_IS_FRESH_INSTALL=1'

https://commcare-cloud.readthedocs.io/en/latest/installation/2-manual-install.html#deploy-commcare-hq-services

Hello Simon,

Thank you for taking the time.

Have you been able to successfully deploy in the past?

I'm not sure but I think so, because the instance works (was used) with the current install release. (September 2020)

If so, do the requirements files exist in the current release?

The contents of the requirements repository of the current release are as follows :

/home/cchq/www/monolith/current/requirements$ ls -a
. build-requirements.txt prod-requirements.in requirements.in test-requirements.txt yum-packages.txt
.. dev-requirements.in prod-requirements.txt requirements.txt uninstall-requirements-after-install.txt
apt-packages.txt dev-requirements.txt rabbitmq-packages.txt test-requirements.in uninstall-requirements.txt

Do I need to run this command since the current release is not a fresh installation ?

commcare-cloud monolith deploy-stack --skip-check -e 'CCHQ_IS_FRESH_INSTALL=1'

No you should not need to run this. However since this installation has not been touched in a long time I would recommend going through the change logs to ensure that all required maintenance has been done.

On specific thing I noticed when re-examining the error is that this environment is still running Python 3.6 which is no longer supported. You can find the change log entry for upgrading Python here: 45. Upgrade CommCare HQ to Python 3.9 — CommCareHQ Deployment documentation

But before you do that I think it would be best to review all the change logs and ensure they are applied in order.

1 Like

I will apply the required maintenance in changelog.

After that I will re-try deploy command.

I will keep you informed about my progress.

Thank you,

Hi Simon,

I notice an warning when connecting to the commcare.

To finish first-time installation, run "manage-commcare-cloud configure"

Is this normal?

Hi @Simon_Kelly,

I have followed your recommendations by applying the changelog instructions.

I'm on 2020-10-14 Run command to update Supervisor configurations.

The instruction is to first deploy the following version of CommCare 05ebc51b, which I did by executing :

$ commcare-cloud monolith deploy --commcare-rev 05ebc51ba4b16a420f0fd64c17ffad89501ea36d

I encounter the following error :

...
[x.x.x.x] sudo: git reset --hard 05ebc51ba4b16a420f0fd64c17ffad89501ea36d
[x.x.x.x] out: HEAD est maintenant Ă  05ebc51ba4b Merge pull request #28632 from dimagi/dm/rm-errand-boy
[x.x.x.x] out:

[x.x.x.x] sudo: git submodule sync
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'corehq/apps/hqmedia/static/hqmedia/MediaUploader'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/commcare-translations'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/couchdbkit-aggregate'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/django-digest-src'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/django-no-exceptions'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/langcodes'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/python-digest'
[x.x.x.x] out: Synchronisation de l'URL sous-module pour 'submodules/xml2json'
[x.x.x.x] out:

[x.x.x.x] sudo: git submodule update --init --recursive -q
[x.x.x.x] out: fatal: impossible de se connecter Ă  github.com :
[x.x.x.x] out: github.com[0: 140.82.121.4]: errno=Connexion terminée par expiration du délai d'attente
[x.x.x.x] out:
[x.x.x.x] out: fatal: impossible de se connecter Ă  github.com :
[x.x.x.x] out: github.com[0: 140.82.121.3]: errno=Connexion terminée par expiration du délai d'attente
[x.x.x.x] out:
[x.x.x.x] out: Chemin de sous-module 'submodules/langcodes' récupéré, mais il ne contenait pas 170e74cf47c4757502bc1654414930685aacaa5a. La récupération directe de ce commit a échoué.
[x.x.x.x] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: git submodule update --init --recursive -q
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2022-07-05_09.21 >/dev/null && git submodule update --init --recursive -q"

Aborting.
!!! Parallel execution exception under host u'x.x.x.x':
Process x.x.x.x:
...
Underlying exception:
sudo() received nonzero return code 1 while executing!

Requested: git submodule update --init --recursive -q
Executed: sudo -S -p '\"'\"'sudo password:'\"'\"' -H  -u \"cchq\"  /bin/bash -l -c \"cd /home/cchq/www/monolith/releases/2022-07-05_09.21 >/dev/null && git submodule update --init --recursive -q\"' --environment monolith"

...

It seems that he was trying to get a commit that does not exist (170e74cf47c4757502bc1654414930685aacaa5a).

Does this mean anything to you?

Hi, the commit definitely does exist (Add Nuer · dimagi/langcodes@170e74c · GitHub). I think the issue may be related to the errors above that which are connection timeouts connecting to Github.

You could try running the command manually to see if the error persists:

sudo -iu cchq
cd /home/cchq/www/monolith/releases/2022-07-05_09.21
git submodule update --init --recursive
1 Like

Hello,

Still have the same error :

(monolith) cchq@monolith:~$ cd /home/cchq/www/monolith/releases/2022-07-05_09.21
(monolith) cchq@monolith:~/www/monolith/releases/2022-07-05_09.21$ git submodule update --init --recursive
fatal: impossible de se connecter Ă  github.com :
github.com[0: 140.82.121.3]: errno=Connexion terminée par expiration du délai d'attente

Impossible de rapatrier dans le chemin de sous-module 'submodules/langcodes' ; essai de rapatriement direct de 170e74cf47c4757502bc1654414930685aacaa5a :
fatal: impossible de se connecter Ă  github.com :
github.com[0: 140.82.121.3]: errno=Connexion terminée par expiration du délai d'attente

Chemin de sous-module 'submodules/langcodes' récupéré, mais il ne contenait pas 170e74cf47c4757502bc1654414930685aacaa5a. La récupération directe de ce commit a échoué.

What do you see if you run this?

$ git submodule foreach 'git remote -v'

That looks like the issue @jmiller found here

1 Like

Hello Ethan & @Simon_Kelly,
Sorry for the delay. I had an issue with my hoster..

Command execution on release i try to deploy :

(monolith) cchq@monolith:~/www/monolith/releases/2022-07-15_12.13$ git submodule foreach 'git remote -v'
Entrée dans 'corehq/apps/hqmedia/static/hqmedia/MediaUploader'
origin git://github.com/dimagi/MediaUploader.git (fetch)
origin git://github.com/dimagi/MediaUploader.git (push)
Entrée dans 'submodules/commcare-translations'
origin git://github.com/dimagi/commcare-translations.git (fetch)
origin git://github.com/dimagi/commcare-translations.git (push)
Entrée dans 'submodules/couchdbkit-aggregate'
origin git://github.com/dimagi/couchdbkit-aggregate.git (fetch)
origin git://github.com/dimagi/couchdbkit-aggregate.git (push)
Entrée dans 'submodules/django-digest-src'
origin git://github.com/dimagi/django-digest.git (fetch)
origin git://github.com/dimagi/django-digest.git (push)
Entrée dans 'submodules/django-no-exceptions'
origin git://github.com/dimagi/django-no-exceptions.git (fetch)
origin git://github.com/dimagi/django-no-exceptions.git (push)
Entrée dans 'submodules/langcodes'
origin git://github.com/dimagi/langcodes.git (fetch)
origin git://github.com/dimagi/langcodes.git (push)
Entrée dans 'submodules/python-digest'
origin git://github.com/dimagi/python-digest.git (fetch)
origin git://github.com/dimagi/python-digest.git (push)
Entrée dans 'submodules/xml2json'
origin git://github.com/dimagi/xml2json.git (fetch)
origin git://github.com/dimagi/xml2json.git (push)

This is surprising because I have changed the format of the url in the current version as you can see here to correct the issue identified in the post you quoted (That looks like the issue @jmiller found here):

(monolith) cchq@monolith:~/www/monolith/current$ git submodule foreach 'git remote -v'
Entrée dans 'corehq/apps/hqmedia/static/hqmedia/MediaUploader'
origin https://github.com/dimagi/MediaUploader.git(fetch)
origin https://github.com/dimagi/MediaUploader.git(push)
Entrée dans 'submodules/commcare-translations'
origin https://github.com/dimagi/commcare-translations.git(fetch)
origin https://github.com/dimagi/commcare-translations.git(push)
Entrée dans 'submodules/couchdbkit-aggregate'
origin https://github.com/dimagi/couchdbkit-aggregate.git(fetch)
origin https://github.com/dimagi/couchdbkit-aggregate.git(push)
Entrée dans 'submodules/django-digest-src'
origin https://github.com/dimagi/django-digest.git(fetch)
origin https://github.com/dimagi/django-digest.git(push)
Entrée dans 'submodules/django-no-exceptions'
origin https://github.com/dimagi/django-no-exceptions.git(fetch)
origin https://github.com/dimagi/django-no-exceptions.git(push)
Entrée dans 'submodules/langcodes'
origin https://github.com/dimagi/langcodes.git(fetch)
origin https://github.com/dimagi/langcodes.git(push)
Entrée dans 'submodules/python-digest'
origin https://github.com/dimagi/python-digest.git(fetch)
origin https://github.com/dimagi/python-digest.git(push)
Entrée dans 'submodules/xml2json'
origin https://github.com/dimagi/xml2json.git(fetch)
origin https://github.com/dimagi/xml2json.git(push)

I modified the url format in the release /home/cchq/www/monolith/releases/2022-07-15_12.13 then run the command

After that, i try to resume the deploy with

(monolith) it@monolith:~$ commcare-cloud monolith deploy --resume --commcare-rev 05ebc51ba4b16a420f0fd64c17ffad89501ea36d

but occur this error :

...
[x.x.x.x] sudo: git config submodule.submodules/xm> l2json.url
[x.x.x.x] out: git://github.com/dimagi/xml2json.git
[x.x.x.x] out:

[x.x.x.x] sudo: git clone /home/cchq/www/monolith/current/.git /home/cchq/www/monolith/releases/2022-07-15_12.13
[x.x.x.x] out: fatal: le chemin de destination '/home/cchq/www/monolith/releases/2022-07-15_12.13' existe déjà et n'est pas un répertoire vide.
[x.x.x.x] out:

Fatal error: sudo() received nonzero return code 128 while executing!

Requested: git clone /home/cchq/www/monolith/current/.git /home/cchq/www/monolith/releases/2022-07-15_12.13
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/monolith/current >/dev/null && git clone /home/cchq/www/monolith/current/.git /home/cchq/www/monolith/releases/2022-07-15_12.13"

Aborting.
!!! Parallel execution exception under host u'x.x.x.x':
...
Underlying exception:
sudo() received nonzero return code 128 while executing!

Requested: git clone /home/cchq/www/monolith/current/.git /home/cchq/www/monolith/releases/2022-07-15_12.13
Executed: sudo -S -p '\"'\"'sudo password:'\"'\"' -H  -u \"cchq\"  /bin/bash -l -c \"cd /home/cchq/www/monolith/current >/dev/null && git clone /home/cchq/www/monolith/current/.git /home/cchq/www/monolith/releases/2022-07-15_12.13\"' --environment monolith"

Disconnecting from x.x.x.x... done.

How to correctly resume deploy on this release ? (2022-07-15_12.13)

I'm not sure it's worth resuming the deploy - have you tried just doing a new one entirely?

There should be multiple copies of the git repository inside ~/www/monolith/releases/, one for each recent release (the deploy script deletes the older ones eventually). When a deploy is run, it copies the currently active repo to a new directory. I suspect what may have happened is you correctly updated the URLs for the new directory, but not for the old one, then on redeploy, it overwrote it again.

I'd make sure to update the URLs for the current repo (the one aliased to ~/www/monolith/current/), then try a new deploy without using --resume.

1 Like

I'm not sure it's worth resuming the deploy - have you tried just doing a new one entirely?

Yes i tried it, but I have the same error.

When deploy --resume is used, it proceeds with the failed deployment. Does this also apply to a failed deploy with the --commcare-rev argument?

Here is the urls on the current release.

(monolith) it@monolith:/home/cchq/www/monolith/current$ git remote -v
origin https://github.com/dimagi/commcare-hq.git (fetch)
origin https://github.com/dimagi/commcare-hq.git (push)
(monolith) it@monolith:/home/cchq/www/monolith/current$ git submodule foreach 'git remote -v'
Entrée dans 'corehq/apps/hqmedia/static/hqmedia/MediaUploader'
origin https://github.com/dimagi/MediaUploader.git (fetch)
origin https://github.com/dimagi/MediaUploader.git (push)
Entrée dans 'submodules/commcare-translations'
origin https://github.com/dimagi/commcare-translations.git (fetch)
origin https://github.com/dimagi/commcare-translations.git (push)
Entrée dans 'submodules/couchdbkit-aggregate'
origin https://github.com/dimagi/couchdbkit-aggregate.git (fetch)
origin https://github.com/dimagi/couchdbkit-aggregate.git (push)
Entrée dans 'submodules/django-digest-src'
origin https://github.com/dimagi/django-digest.git (fetch)
origin https://github.com/dimagi/django-digest.git (push)
Entrée dans 'submodules/django-no-exceptions'
origin https://github.com/dimagi/django-no-exceptions.git (fetch)
origin https://github.com/dimagi/django-no-exceptions.git (push)
Entrée dans 'submodules/langcodes'
origin https://github.com/dimagi/langcodes.git (fetch)
origin https://github.com/dimagi/langcodes.git (push)
Entrée dans 'submodules/python-digest'
origin https://github.com/dimagi/python-digest.git (fetch)
origin https://github.com/dimagi/python-digest.git (push)
Entrée dans 'submodules/xml2json'
origin https://github.com/dimagi/xml2json.git (fetch)
origin https://github.com/dimagi/xml2json.git (push)

Are they all correct?

I don't understand why in the release I want to deploy (05ebc51ba4b16a420f0fd64c17ffad89501ea36d) the urls are not correct although the urls are correct in the current version.
I have suspicion that the git clone of the current release when deploying is not working properly, or that there is some other problem.
Does this make sense?

@Ethan_Soergel @Simon_Kelly @jmiller

I post here the full logs of the failed deploy.

I thought there is an issue with the submodules urls on the release to be deployed.

The current release submodule urls looks right to me.

After the git clone and the following commands the release to be deployed has no more correct submodules urls.

What could be the origin of this?

Thanks in advance.

I do see this in the logs for each submodule:

.gitmodules, multiple configurations found for 'submodule.corehq/apps/hqmedia/static/hqmedia/MediaUploader.path'. Skipping second one!

I wonder if somehow the submodules are configured twice each, with different URL formats. Not sure how that could've happened, but I suspect you'll need to correct that manually.

1 Like

Which configuration (file) are you referring to?
Can you lead me a bit further?
Thanks

This is the contain of .gitmodules on the current release (/home/cchq/www/monolith/current):

[submodule "corehq/apps/hqmedia/static/hqmedia/MediaUploader"]
path = corehq/apps/hqmedia/static/hqmedia/MediaUploader
url = https://github.com/dimagi/MediaUploader.git
[submodule "submodules/commcare-translations"]
path = submodules/commcare-translations
url = https://github.com/dimagi/commcare-translations.git
[submodule "submodules/couchdbkit-aggregate"]
path = submodules/couchdbkit-aggregate
url = https://github.com/dimagi/couchdbkit-aggregate.git
[submodule "submodules/django-digest-src"]
path = submodules/django-digest-src
url = https://github.com/dimagi/django-digest.git
[submodule "submodules/django-no-exceptions"]
path = submodules/django-no-exceptions
url = https://github.com/dimagi/django-no-exceptions.git
[submodule "submodules/langcodes"]
path = submodules/langcodes
url = https://github.com/dimagi/langcodes.git
[submodule "submodules/python-digest"]
path = submodules/python-digest
url = https://github.com/dimagi/python-digest.git
[submodule "submodules/xml2json"]
path = submodules/xml2json
url = https://github.com/dimagi/xml2json.git

Is there anything wrong ?

The .gitmodules file of the version to deploy (which failed) has an incorrect url format: git://

Actually I might've been mistaken about that "multiple configurations" message - I see that as well when I deploy.

The .gitmodules file of the version to deploy (which failed) has an incorrect url format: git://

Honestly I'm not sure how to proceed from here - it must be getting those old URL formats from somewhere, I'd try to figure out where.

1 Like