Deploy failing - log included

A cchq monolith deploy and deploy formplayer are both failing. Output for the deploy:

https://pastebin.com/raw/TFrxhqHx

Any suggestions appreciated! It must be noted that this system has been out of date for a while - the last deploy appears to have been done using 2019-08-02_10.30.
I'm off to double check the changelog to see what may have been missed...

Hi Ed

This is a tricky one, I think what has likely happened is that the localsettings.py file is out of sync with the code.

The simplest thing might actually be to set up CommCare on the machine from scratch. You could do this by renaming the /home/cchq/monolith directory to /home/cchq/monolith_backup and then running Ansible to re-create the setup:

cchq monolith ap deploy_commcarehq.yml
1 Like

Thanks Simon - quick one, did you mean the /home/cchq/www/monolith directory?
Assuming that, it's erroring out on the pip requirements installation.

TASK [commcarehq : install pip requirements (https://github.com/dimagi/commcare-hq.git)] ****************************************************************************************************
fatal: [10.0.0.4]: FAILED! => {"changed": false, "cmd": ["/home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6/bin/pip", "install", "-r", "/home/cchq/www/monolith/releases/2021-09-15_07.15/requirements/prod-requirements.txt"], "msg": "stdout: Already using interpreter /usr/bin/python3.6\nUsing base prefix '/usr'\nNew python executable in /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6/bin/python3.6\nAlso creating executable in /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6/bin/python\nInstalling setuptools, pip, wheel...\ndone.\nRunning virtualenv with interpreter /usr/bin/python3.6\nCollecting celery==4.1.1.1\n  Downloading https://github.com/dimagi/celery/raw/4e4ad229423db5568186c94fdbfaa00f0d5b76c9/releases/celery-4.1.1.1-py2.py3-none-any.whl (400 kB)\nCollecting jsonpath-ng==1.5.2.2\n  Downloading https://github.com/kaapstorm/python-jsonpath-rw/raw/wherenot+find_or_create/wheel/jsonpath_ng-1.5.2.2-py3-none-any.whl (29 kB)\nCollecting alembic==1.4.3\n  Downloading alembic-1.4.3-py2.py3-none-any.whl (159 kB)\nCollecting amqp==2.6.1\n  Downloading amqp-2.6.1-py2.py3-none-any.whl (48 kB)\nCollecting architect==0.5.6\n  Downloading architect-0.5.6-py2.py3-none-any.whl (31 kB)\nCollecting attrs==18.2.0\n  Using cached attrs-18.2.0-py2.py3-none-any.whl (34 kB)\nCollecting babel==2.8.0\n  Downloading Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)\nCollecting backcall==0.2.0\n  Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)\nCollecting billiard==3.5.0.4\n  Using cached billiard-3.5.0.4.tar.gz (150 kB)\nCollecting boto3==1.17.85\n  Downloading boto3-1.17.85-py2.py3-none-any.whl (131 kB)\nCollecting botocore==1.20.85\n  Downloading botocore-1.20.85-py2.py3-none-any.whl (7.6 MB)\nCollecting certifi==2020.6.20\n  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)\nCollecting cffi==1.14.3\n  Downloading cffi-1.14.3-cp36-cp36m-manylinux1_x86_64.whl (400 kB)\nCollecting chardet==3.0.4\n  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)\nCollecting cloudant==2.14.0\n  Downloading cloudant-2.14.0.tar.gz (61 kB)\nCollecting commcaretranslationchecker==0.9.7\n  Downloading CommcareTranslationChecker-0.9.7.tar.gz (15 kB)\nCollecting concurrent-log-handler==0.9.12\n  Using cached concurrent_log_handler-0.9.12-py2.py3-none-any.whl (41 kB)\nCollecting contextlib2==0.6.0.post1\n  Downloading contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)\nCollecting cryptography==3.4.8\n  Downloading cryptography-3.4.8-cp36-abi3-manylinux_2_24_x86_64.whl (3.0 MB)\nCollecting csiphash==0.0.5\n  Using cached csiphash-0.0.5.tar.gz (12 kB)\nCollecting datadog==0.39.0\n  Downloading datadog-0.39.0-py2.py3-none-any.whl (106 kB)\nCollecting ddtrace==0.44.0\n  Downloading ddtrace-0.44.0-cp36-cp36m-manylinux2010_x86_64.whl (2.1 MB)\nCollecting decorator==4.0.11\n  Using cached decorator-4.0.11-py2.py3-none-any.whl (8.9 kB)\nCollecting defusedxml==0.5.0\n  Using cached defusedxml-0.5.0-py2.py3-none-any.whl (33 kB)\nCollecting deprecated==1.2.10\n  Downloading Deprecated-1.2.10-py2.py3-none-any.whl (8.7 kB)\nCollecting diff-match-patch==20200713\n  Downloading diff_match_patch-20200713-py3-none-any.whl (61 kB)\nCollecting dimagi-memoized==1.1.3\n  Downloading dimagi_memoized-1.1.3-py2.py3-none-any.whl (3.7 kB)\nCollecting django==2.2.24\n  Downloading Django-2.2.24-py3-none-any.whl (7.5 MB)\nCollecting django-angular==2.2.4\n  Downloading django-angular-2.2.4.tar.gz (165 kB)\nCollecting django-appconf==1.0.4\n  Downloading django_appconf-1.0.4-py2.py3-none-any.whl (6.3 kB)\nCollecting django-autoslug==1.9.8\n  Downloading django_autoslug-1.9.8-py2.py3-none-any.whl (14 kB)\nCollecting django-braces==1.14.0\n  Downloading django_braces-1.14.0-py2.py3-none-any.whl (14 kB)\nCollecting django-bulk-update==2.2.0\n  Using cached django_bulk_update-2.2.0-py2.py3-none-any.whl (19 kB)\nCollecting django-celery-results==1.0.4\n  Using cached django_celery_results-1.0.4-py2.py3-none-any.whl (16 kB)\nCollecting django-compressor==2.4\n  Downloading django_compressor-2.4-py2.py3-none-any.whl (126 kB)\nCollecting django-countries==4.6\n  Using cached django_countries-4.6-py2.py3-none-any.whl (627 kB)\nCollecting django-crispy-forms==1.10.0\n  Downloading django_crispy_forms-1.10.0-py3-none-any.whl (107 kB)\nCollecting django-cte==1.1.4\n  Using cached django_cte-1.1.4-py2.py3-none-any.whl (9.8 kB)\nCollecting django-formtools==2.1\n  Using cached django_formtools-2.1-py2.py3-none-any.whl (132 kB)\nCollecting django-logentry-admin==1.0.6\n  Downloading django_logentry_admin-1.0.6-py3-none-any.whl (9.0 kB)\nCollecting django-oauth-toolkit==1.5.0\n  Downloading django_oauth_toolkit-1.5.0-py3-none-any.whl (60 kB)\nCollecting django-otp==0.9.4\n  Downloading django_otp-0.9.4-py3-none-any.whl (53 kB)\nCollecting django-partial-index==0.6.0\n  Downloading django_partial_index-0.6.0-py2.py3-none-any.whl (11 kB)\nCollecting django-phonenumber-field==2.3.1\n  Downloading django_phonenumber_field-2.3.1-py2.py3-none-any.whl (45 kB)\nCollecting django-prbac==1.0.1\n  Downloading django_prbac-1.0.1-py2.py3-none-any.whl (18 kB)\nCollecting django-ranged-response==0.2.0\n  Using cached django-ranged-response-0.2.0.tar.gz (3.0 kB)\nCollecting django-redis==4.12.1\n  Downloading django_redis-4.12.1-py3-none-any.whl (19 kB)\nCollecting django-redis-sessions==0.6.1\n  Using cached django-redis-sessions-0.6.1.tar.gz (10 kB)\nCollecting django-simple-captcha==0.5.13\n  Downloading django-simple-captcha-0.5.13.zip (166 kB)\nCollecting django-statici18n==1.9.0\n  Downloading django_statici18n-1.9.0-py2.py3-none-any.whl (9.4 kB)\nCollecting django-tastypie==0.14.3\n  Downloading django-tastypie-0.14.3.tar.gz (148 kB)\nCollecting django-transfer==0.4\n  Using cached django-transfer-0.4.tar.gz (16 kB)\nCollecting django-two-factor-auth==1.12.1\n  Downloading django_two_factor_auth-1.12.1-py2.py3-none-any.whl (180 kB)\nCollecting django-user-agents==0.4.0\n  Downloading django_user_agents-0.4.0-py3-none-any.whl (8.6 kB)\nCollecting django-websocket-redis==0.6.0\n  Downloading django-websocket-redis-0.6.0.tar.gz (19 kB)\nCollecting djangorestframework==3.12.2\n  Downloading djangorestframework-3.12.2-py3-none-any.whl (957 kB)\nCollecting dnspython==1.15.0\n  Using cached dnspython-1.15.0-py2.py3-none-any.whl (177 kB)\nCollecting dropbox==9.3.0\n  Using cached dropbox-9.3.0-py3-none-any.whl (503 kB)\nCollecting elasticsearch2==2.5.1\n  Downloading elasticsearch2-2.5.1-py2.py3-none-any.whl (57 kB)\nCollecting elasticsearch5==5.5.6\n  Downloading elasticsearch5-5.5.6-py2.py3-none-any.whl (62 kB)\nCollecting email_validator==1.1.3\n  Downloading email_validator-1.1.3-py2.py3-none-any.whl (18 kB)\nCollecting et-xmlfile==1.0.1\n  Using cached et_xmlfile-1.0.1.tar.gz (8.4 kB)\nCollecting ethiopian-date-converter==0.1.5\n  Using cached ethiopian-date-converter-0.1.5.tar.gz (3.7 kB)\nCollecting eulxml==1.1.3\n  Using cached eulxml-1.1.3-py2.py3-none-any.whl (216 kB)\nCollecting faker==5.0.2\n  Downloading Faker-5.0.2-py3-none-any.whl (1.1 MB)\nCollecting flower==0.9.2\n  Using cached flower-0.9.2.tar.gz (1.3 MB)\nCollecting gevent==1.4.0\n  Using cached gevent-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (5.5 MB)\nCollecting ghdiff==0.4\n  Using cached ghdiff-0.4.tar.gz (5.7 kB)\nCollecting greenlet==0.4.17\n  Downloading greenlet-0.4.17-cp36-cp36m-manylinux1_x86_64.whl (44 kB)\nCollecting gunicorn==20.0.4\n  Downloading gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)\nCollecting hiredis==2.0.0\n  Downloading hiredis-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (84 kB)\nCollecting httpagentparser==1.9.0\n  Downloading httpagentparser-1.9.0.tar.gz (7.1 kB)\nCollecting idna==2.10\n  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)\nCollecting importlib-metadata==3.4.0\n  Downloading importlib_metadata-3.4.0-py3-none-any.whl (10 kB)\nCollecting intervaltree==3.1.0\n  Downloading intervaltree-3.1.0.tar.gz (32 kB)\nCollecting ipython==7.16.1\n  Downloading ipython-7.16.1-py3-none-any.whl (785 kB)\nCollecting ipython-genutils==0.2.0\n  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)\nCollecting iso8601==0.1.13\n  Downloading iso8601-0.1.13-py3-none-any.whl (9.3 kB)\nCollecting isodate==0.6.0\n  Downloading isodate-0.6.0-py2.py3-none-any.whl (45 kB)\nCollecting jdcal==1.4.1\n  Using cached jdcal-1.4.1-py2.py3-none-any.whl (9.5 kB)\nCollecting jedi==0.17.2\n  Downloading jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)\nCollecting jinja2==2.11.3\n  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)\nCollecting jmespath==0.10.0\n  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)\nCollecting json-delta==2.0\n  Using cached json_delta-2.0-py2.py3-none-any.whl (34 kB)\nCollecting jsonfield==2.1.1\n  Downloading jsonfield-2.1.1-py2.py3-none-any.whl (11 kB)\nCollecting jsonobject==0.9.9\n  Using cached jsonobject-0.9.9.tar.gz (389 kB)\nCollecting jsonobject-couchdbkit==1.0.1\n  Downloading jsonobject_couchdbkit-1.0.1-py2.py3-none-any.whl (56 kB)\nCollecting jsonschema==3.2.0\n  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\nCollecting jwcrypto==0.8\n  Downloading jwcrypto-0.8-py2.py3-none-any.whl (79 kB)\nCollecting kafka-python==1.4.7\n  Downloading kafka_python-1.4.7-py2.py3-none-any.whl (266 kB)\nCollecting kombu==4.2.2.post1\n  Using cached kombu-4.2.2.post1-py2.py3-none-any.whl (177 kB)\nCollecting laboratory==0.2.0\n  Using cached laboratory-0.2.0-py2.py3-none-any.whl (7.6 kB)\nCollecting lxml==4.6.3\n  Downloading lxml-4.6.3-cp36-cp36m-manylinux2014_x86_64.whl (6.3 MB)\nCollecting mako==1.1.3\n  Downloading Mako-1.1.3-py2.py3-none-any.whl (75 kB)\nCollecting markdown==2.2.1\n  Downloading Markdown-2.2.1.zip (362 kB)\nCollecting markupsafe==1.1.1\n  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux2010_x86_64.whl (32 kB)\nCollecting mock==2.0.0\n  Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)\nCollecting ndg-httpsclient==0.5.1\n  Using cached ndg_httpsclient-0.5.1-py3-none-any.whl (34 kB)\nCollecting oauthlib==3.1.0\n  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)\nCollecting openpyxl==3.0.6\n  Downloading openpyxl-3.0.6-py2.py3-none-any.whl (242 kB)\nCollecting parso==0.7.1\n  Downloading parso-0.7.1-py2.py3-none-any.whl (109 kB)\nCollecting pbr==5.5.0\n  Downloading pbr-5.5.0-py2.py3-none-any.whl (106 kB)\nCollecting pexpect==4.8.0\n  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)\nCollecting phonenumberslite==8.12.10\n  Downloading phonenumberslite-8.12.10-py2.py3-none-any.whl (448 kB)\nCollecting pickleshare==0.7.5\n  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)\nCollecting pillow==8.3.2\n  Downloading Pillow-8.3.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\nCollecting ply==3.11\n  Using cached ply-3.11-py2.py3-none-any.whl (49 kB)\nCollecting polib==1.1.0\n  Using cached polib-1.1.0-py2.py3-none-any.whl (25 kB)\nCollecting prometheus-client==0.7.1\n  Downloading prometheus_client-0.7.1.tar.gz (38 kB)\nCollecting prompt-toolkit==3.0.7\n  Downloading prompt_toolkit-3.0.7-py3-none-any.whl (355 kB)\nCollecting protobuf==3.13.0\n  Downloading protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB)\nCollecting psycogreen==1.0.2\n  Downloading psycogreen-1.0.2.tar.gz (5.4 kB)\n  Installing build dependencies: started\n  Installing build dependencies: finished with status 'done'\n  Getting requirements to build wheel: started\n  Getting requirements to build wheel: finished with status 'done'\n    Preparing wheel metadata: started\n    Preparing wheel metadata: finished with status 'done'\nCollecting psycopg2==2.8.6\n  Downloading psycopg2-2.8.6.tar.gz (383 kB)\nCollecting ptyprocess==0.6.0\n  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)\nCollecting py-kissmetrics==1.1.0\n  Downloading py_KISSmetrics-1.1.0-py2.py3-none-any.whl (8.0 kB)\nCollecting pyasn1==0.4.8\n  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)\nCollecting pycparser==2.20\n  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)\nCollecting pycryptodome==3.10.1\n  Downloading pycryptodome-3.10.1-cp35-abi3-manylinux2010_x86_64.whl (1.9 MB)\nCollecting pygithub==1.54.1\n  Downloading PyGithub-1.54.1-py3-none-any.whl (289 kB)\nCollecting pygments==2.8.1\n  Downloading Pygments-2.8.1-py3-none-any.whl (983 kB)\nCollecting pygooglechart==0.4.0\n  Using cached pygooglechart-0.4.0.tar.gz (28 kB)\nCollecting pyjwt==1.7.1\n  Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)\nCollecting pyopenssl==20.0.1\n  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)\nCollecting pyphonetics==0.5.3\n  Downloading pyphonetics-0.5.3-py2.py3-none-any.whl (10 kB)\nCollecting pyrsistent==0.17.3\n  Downloading pyrsistent-0.17.3.tar.gz (106 kB)\nCollecting pysocks==1.7.1\n  Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)\nCollecting python-dateutil==2.8.1\n  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)\nCollecting python-editor==1.0.4\n  Using cached python_editor-1.0.4-py3-none-any.whl (4.9 kB)\nCollecting python-imap==1.0.0\n  Downloading python-imap-1.0.0.tar.gz (763 bytes)\nCollecting python-magic==0.4.22\n  Downloading python_magic-0.4.22-py2.py3-none-any.whl (12 kB)\nCollecting python-mimeparse==1.6.0\n  Using cached python_mimeparse-1.6.0-py2.py3-none-any.whl (6.1 kB)\nCollecting python3-saml==1.9.0\n  Downloading python3_saml-1.9.0-py3-none-any.whl (72 kB)\nCollecting pytz==2020.1\n  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)\nCollecting pyyaml==5.4.1\n  Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)\nCollecting pyzxcvbn==0.8.0\n  Using cached pyzxcvbn-0.8.0-py2.py3-none-any.whl (339 kB)\nCollecting qrcode==4.0.4\n  Using cached qrcode-4.0.4.tar.gz (18 kB)\nCollecting quickcache==0.5.4\n  Using cached quickcache-0.5.4-py2.py3-none-any.whl (7.9 kB)\nCollecting rcssmin==1.0.6\n  Using cached rcssmin-1.0.6.tar.gz (582 kB)\nCollecting redis==3.5.3\n  Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)\nCollecting reportlab==3.5.53\n  Downloading reportlab-3.5.53-cp36-cp36m-manylinux2010_x86_64.whl (2.6 MB)\nCollecting requests==2.25.1\n  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)\nCollecting requests-oauthlib==1.3.0\n  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)\nCollecting requests_toolbelt==0.9.1\n  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\nCollecting rjsmin==1.1.0\n  Downloading rjsmin-1.1.0-cp36-cp36m-manylinux1_x86_64.whl (23 kB)\nCollecting s3transfer==0.4.2\n  Downloading s3transfer-0.4.2-py2.py3-none-any.whl (79 kB)\nCollecting schema==0.7.2\n  Downloading schema-0.7.2-py2.py3-none-any.whl (16 kB)\nCollecting sentry-sdk==0.19.5\n  Downloading sentry_sdk-0.19.5-py2.py3-none-any.whl (128 kB)\nCollecting setproctitle==1.2.2\n  Downloading setproctitle-1.2.2-cp36-cp36m-manylinux1_x86_64.whl (35 kB)\nCollecting sh==1.14.1\n  Downloading sh-1.14.1-py2.py3-none-any.whl (40 kB)\nCollecting simpleeval==0.9.10\n  Downloading simpleeval-0.9.10.tar.gz (26 kB)\n\n:stderr:     ERROR: Command errored out with exit status 1:\n     command: /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/setup.py'\"'\"'; __file__='\"'\"'/tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /tmp/pip-pip-egg-info-l101p280\n         cwd: /tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/\n    Complete output (1 lines):\n    error in simpleeval setup command: use_2to3 is invalid.\n    ----------------------------------------\nWARNING: Discarding https://files.pythonhosted.org/packages/62/25/aec98426834844b70b7ab24b4cce8655d31e654f58e1fa9861533f5f2af1/simpleeval-0.9.10.tar.gz#sha256=692055488c2864637f6c2edb5fa48175978a2a07318009e7cf03c9790ca17bea (from https://pypi.org/simple/simpleeval/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\nERROR: Could not find a version that satisfies the requirement simpleeval==0.9.10 (from versions: 0.8.1.linux-x86_64, 0.9.0.linux-x86_64, 0.9.1.linux-x86_64, 0.8.1, 0.8.2, 0.8.5, 0.8.6, 0.8.7, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.9.10)\nERROR: No matching distribution found for simpleeval==0.9.10\n"}

This looks like the relevant details:

  Downloading simpleeval-0.9.10.tar.gz (26 kB)
  
  :stderr:     ERROR: Command errored out with exit status 1:
     command: /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/setup.py'\"'\"'; __file__='\"'\"'/tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\
     '\"'\"', '\"'\"'\
'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' egg_info --egg-base /tmp/pip-pip-egg-info-l101p280
         cwd: /tmp/pip-install-gva3f0go/simpleeval_2a0639e0f1fc43c9bc6b37b1ca75e6d5/
    Complete output (1 lines):
    error in simpleeval setup command: use_2to3 is invalid.
    ----------------------------------------
    
WARNING: Discarding https://files.pythonhosted.org/packages/62/25/aec98426834844b70b7ab24b4cce8655d31e654f58e1fa9861533f5f2af1/simpleeval-0.9.10.tar.gz#sha256=692055488c2864637f6c2edb5fa48175978a2a07318009e7cf03c9790ca17bea (from https://pypi.org/simple/simpleeval/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement simpleeval==0.9.10 (from versions: 0.8.1.linux-x86_64, 0.9.0.linux-x86_64, 0.9.1.linux-x86_64, 0.8.1, 0.8.2, 0.8.5, 0.8.6, 0.8.7, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.9.10)
ERROR: No matching distribution found for simpleeval==0.9.10
"}

What's odd is that it can't seem to find simpleeval 0.9.10 when it appears to be on that list...

There was a breaking change in the setuptools package that's causing this error. This will require a config change on our side. I'll dig into that and get back to you.

1 Like

I did some local testing and I'm not completely sure how to resolve this. In the mean time I got it working locally using this workaround:

  1. activate the virtual environment in the release

    source python_env-3.6/bin/activate
    
  2. downgrade setuptools

    pip install "setuptools<58"
    
  3. install the requirements

    pip install -r requirements/prod-requirements.txt
    

After that the install files will be cached on the machine and you should be able to re-run the commcare-cloud command.
```

1 Like

Great, that seemed to work, off to check that all changes in the changelog have been properly implemented.
Thanks again for the help Simon!

Some things weren't working well after that, so I tried a full deploy using
cchq monolith deploy

That's currently failing on what appears to be an issue related to couchdb during task 'migrate':

https://pastebin.com/raw/AzaraSB9

I see this:
[10.0.0.4] out: django.core.exceptions.FieldError: Cannot resolve keyword 'couch_id' into field. Choices are: actionconfig, alertconfig, consumptionconfig, domain, id, individual_consumption_defaults, stocklevelsconfig, stockrestoreconfig, sync_consumption_fixtures, use_auto_consumption, use_auto_emergency_levels

Hi Ed

It seems the migration and the code have become out of sync. To resolve this you'll need to checkout an older version of the code and then run the migrations:

git checkout 539a7399b995e03fd9ead14ca64a8e66b9b576a4
python manage.py migrate_multi
1 Like

Thanks for that explanation Simon. I think I understand the core issue(s) I've had with these two out of sync instances.

Just to double check what's the ideal place to do this - can I just use the latest directory under /home/cchq/www/releases and run the checkout there? Should I be running it as the cchq user, or is there a script I should be using to access the correct environment?
Thanks again!

EDIT I sourced the /python_env-3.6/bin/activate as root and have this after running migrate_multi:

Traceback (most recent call last):
  File "manage.py", line 194, in <module>
    main()
  File "manage.py", line 41, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "manage.py", line 147, in _setup_once
    _setup_once.setup(*args, **kw)
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/python_env-3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/corehq/apps/cloudcare/models.py", line 13, in <module>
    from corehq.apps.app_manager.models import Application
  File "/home/cchq/www/monolith/releases/2021-09-15_15.13/corehq/apps/app_manager/models.py", line 38, in <module>
    from jsonpath_rw import jsonpath, parse
ModuleNotFoundError: No module named 'jsonpath_rw'

Hi Ed

My apologies, I should have suggested creating a new release to make sure that the python libraries match the code at that point in time:

cchq monolith fab setup_limited_release --set code_branch=539a7399b995e03fd9ead14ca64a8e66b9b576a4
cchq monolith django-manage --release <release created by previous command> migrate_multi

Thanks Simon. Unfortunately the limited release is failing - let me know if you want the full log, but it seems to go awry from here:

Cloning virtual env
[10.0.0.4] sudo: readlink -f /home/cchq/www/monolith/current/python_env-3.6
[10.0.0.4] out: /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6
[10.0.0.4] out:

[10.0.0.4] sudo: virtualenv-clone /home/cchq/www/monolith/releases/2021-09-15_07.15/python_env-3.6 /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6
[10.0.0.4] sudo: sed -i -e "s~/home/cchq/www/monolith/current/python_env-3.6~/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6~g" $(find /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/ -type f)
[10.0.0.4] sudo: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip install --quiet --upgrade --timeout=60 pip-tools
[10.0.0.4] sudo: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt
[10.0.0.4] out:     ERROR: Command errored out with exit status 1:
[10.0.0.4] out:      command: /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n0catb2f/pystache_d1801855c7b24382a6775cd42327d13e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n0catb2f/pystache_d1801855c7b24382a6775cd42327d13e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-fbqj2f3s
[10.0.0.4] out:          cwd: /tmp/pip-install-n0catb2f/pystache_d1801855c7b24382a6775cd42327d13e/
[10.0.0.4] out:     Complete output (2 lines):
[10.0.0.4] out:     pystache: using: version '58.0.4' of <module 'setuptools' from '/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/setuptools/__init__.py'>
[10.0.0.4] out:     error in pystache setup command: use_2to3 is invalid.
[10.0.0.4] out:     ----------------------------------------
[10.0.0.4] out: WARNING: Discarding https://files.pythonhosted.org/packages/d6/fd/eb8c212053addd941cc90baac307c00ac246ac3fce7166b86434c6eae963/pystache-0.5.4.tar.gz#sha256=f7bbc265fb957b4d6c7c042b336563179444ab313fb93a719759111eabd3b85a (from https://pypi.org/simple/pystache/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
[10.0.0.4] out: ERROR: Could not find a version that satisfies the requirement pystache==0.5.4 (from versions: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4)
[10.0.0.4] out: ERROR: No matching distribution found for pystache==0.5.4
[10.0.0.4] out: Traceback (most recent call last):
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/pip-sync", line 8, in <module>
[10.0.0.4] out:     sys.exit(cli())
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/click/core.py", line 1137, in __call__
[10.0.0.4] out:     return self.main(*args, **kwargs)
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/click/core.py", line 1062, in main
[10.0.0.4] out:     rv = self.invoke(ctx)
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/click/core.py", line 1404, in invoke
[10.0.0.4] out:     return ctx.invoke(self.callback, **ctx.params)
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/click/core.py", line 763, in invoke
[10.0.0.4] out:     return __callback(*args, **kwargs)
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/piptools/scripts/sync.py", line 177, in cli
[10.0.0.4] out:     python_executable=python_executable,
[10.0.0.4] out:   File "/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/lib/python3.6/site-packages/piptools/sync.py", line 259, in sync
[10.0.0.4] out:     check=True,
[10.0.0.4] out:   File "/usr/lib/python3.6/subprocess.py", line 438, in run
[10.0.0.4] out:     output=stdout, stderr=stderr)
[10.0.0.4] out: subprocess.CalledProcessError: Command '['/home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/python3.6', '-m', 'pip', 'install', '-r', '/tmp/tmpcsr76_nd', '-q', '--timeout=60']' returned non-zero exit status 1.
[10.0.0.4] out:


Fatal error: sudo() received nonzero return code 1 while executing!

Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt
Executed: sudo -S -p 'sudo password:' -H  -u "cchq"  /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2021-09-16_10.52 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt"

Aborting.
!!! Parallel execution exception under host '10.0.0.4':
Process 10.0.0.4:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 240, in inner
    submit(task.run(*args, **kwargs))
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
    return func(*args, **kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 180, in inner
    return func(*args, **kwargs)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/operations/release.py", line 199, in update
    cmd_prefix, proxy, " ".join(requirements_files)))
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
    stderr=stderr, timeout=timeout, shell_escape=shell_escape,
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
    error(message=msg, stdout=out, stderr=err)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
    return func(message)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
    raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!

Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt
Executed: sudo -S -p 'sudo password:' -H  -u "cchq"  /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2021-09-16_10.52 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt"

Fatal error: One or more hosts failed while executing task 'update'

Underlying exception:
    sudo() received nonzero return code 1 while executing!

    Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt
    Executed: sudo -S -p 'sudo password:' -H  -u "cchq"  /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2021-09-16_10.52 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt"

Aborting.
Traceback (most recent call last):
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
    *args, **kwargs
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
    multiprocessing
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
    return task.run(*args, **kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 288, in setup_limited_release
    _setup_release(parse_int_or_exit(keep_days), full_cluster=False)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 336, in _setup_release
    execute_with_timing(release.update_virtualenv(full_cluster))
  File "/home/ccc/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
    execute(fn, *args, **kwargs)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 417, in execute
    error(err, exception=d['results'])
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
    return func(message)
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
    raise env.abort_exception(msg)
Exception: One or more hosts failed while executing task 'update'

Underlying exception:
    sudo() received nonzero return code 1 while executing!

    Requested: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt
    Executed: sudo -S -p 'sudo password:' -H  -u "cchq"  /bin/bash -l -c "cd /home/cchq/www/monolith/releases/2021-09-16_10.52 >/dev/null && export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-16_10.52/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-16_10.52/requirements/prod-requirements.txt"

That seems to be another issue with user_2to3 but this time for the pystache library. I'm not sure where that dependency is coming from but following the same as above might work:

(replace the last command with pip install pystache

Thanks Simon, I'm getting this on the final command (pip install -r requirements/prod-requirements.txt):

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pip-tools 6.2.0 requires pip>=20.3, but you have pip 20.2.3 which is incompatible.
Successfully installed boto3-1.15.12 botocore-1.18.12 cairocffi-1.1.0 cairosvg-2.5.1 cryptography-3.2 cssselect2-0.3.0 django-2.2.16 django-compressor-2.1 django-oauth-toolkit-1.3.2 elasticsearch-1.9.0 elasticsearch7-7.9.1 email-validator-1.1.2 gipc-1.1.0 hiredis-1.1.0 html5lib-1.1 jinja2-2.11.2 jsonpath-rw-1.4.0 lxml-4.3.5 openpyxl-2.6.4 pillow-7.2.0 pip-20.2.3 pycco-0.5.1 pycryptodome-3.9.9 pygments-2.7.1 pyphen-0.9.5 python-levenshtein-0.12.0 python-magic-0.4.18 pyyaml-5.3.1 requests-2.24.0 rjsmin-1.0.12 s3transfer-0.3.3 sentry-sdk-0.19.4 setproctitle-1.1.10 setuptools-50.3.0 six-1.15.0 smartypants-2.0.1 tinycss2-1.0.2 urllib3-1.25.10 weasyprint-51 webencodings-0.5.1 xmlsec-1.3.9

What's odd is if I do a pip --version, I'm getting:
pip 21.2.4 from /home/cchq/www/monolith/releases/2021-09-17_12.37/python_env-3.6/lib/python3.6/site-packages/pip (python 3.6)

EDIT full log here:

I think that's the issue. Our requirements file of the git version you are using has pip pinned to 20.3 so it's trying to downgrade pip but pip-tools is installed and requires a newer version.

You could try uninstall pip-tool: pip uninstall pip-tools.

Thanks Simon - that seemed to go through fine, though I seem to have lost the momentum here somewhere...
How do I go about creating the new release after resolving those last couple of glitches?
(Going back to your post here: Deploy failing - log included - #11 by Simon_Kelly)
I'm still getting this with the setup_limited_release command:

[10.0.0.4] sudo: export HOME=/home/cchq && source /home/cchq/www/monolith/releases/2021-09-17_14.12/python_env-3.6/bin/activate &&  pip-sync --quiet --pip-args='--timeout=60' /home/cchq/www/monolith/releases/2021-09-17_14.12/requirements/prod-requirements.txt
[10.0.0.4] out:     ERROR: Command errored out with exit status 1:
[10.0.0.4] out:      command: /home/cchq/www/monolith/releases/2021-09-17_14.12/python_env-3.6/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3slqezqo/pystache_72f4d5ed59064b669025f0762ee16506/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3slqezqo/pystache_72f4d5ed59064b669025f0762ee16506/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-acztmh90
[10.0.0.4] out:          cwd: /tmp/pip-install-3slqezqo/pystache_72f4d5ed59064b669025f0762ee16506/
[10.0.0.4] out:     Complete output (2 lines):
[10.0.0.4] out:     pystache: using: version '58.0.4' of <module 'setuptools' from '/home/cchq/www/monolith/releases/2021-09-17_14.12/python_env-3.6/lib/python3.6/site-packages/setuptools/__init__.py'>
[10.0.0.4] out:     error in pystache setup command: use_2to3 is invalid.
[10.0.0.4] out:     ----------------------------------------
[10.0.0.4] out: WARNING: Discarding https://files.pythonhosted.org/packages/d6/fd/eb8c212053addd941cc90baac307c00ac246ac3fce7166b86434c6eae963/pystache-0.5.4.tar.gz#sha256=f7bbc265fb957b4d6c7c042b336563179444ab313fb93a719759111eabd3b85a (from https://pypi.org/simple/pystache/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
[10.0.0.4] out: ERROR: Could not find a version that satisfies the requirement pystache==0.5.4 (from versions: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4)

Thanks again!