Related to this: Process for migrating out of date server - #2 by Simon_Kelly
I am performing the server update mentioned there. I'm currently getting the old server up to date before moving the data.
HQ was at 0b61f715dbcb265d6714f12e0e81e7859c811330 (Thurs Aug 26 2021) at the start
and Cloud was at 185eedf999b14416f2406e55d67e9f97ad24aee5 (Fri Nov 12 2021).
I've successfully performed the following updates from the changelog:
https://commcare-cloud.readthedocs.io/en/latest/changelog/0045-upgrade-to-python-39.html
https://commcare-cloud.readthedocs.io/en/latest/changelog/0046-upgrade_node_14x_npm_7x.html
https://commcare-cloud.readthedocs.io/en/latest/changelog/0048-update-supervisor-confs.html
https://commcare-cloud.readthedocs.io/en/latest/changelog/0052-upgrade-ansible.html
I am now wanting to deploy HQ at the last production deploy before changelog 53 to run the changelog on a more up to date HQ code base but I'm getting the following error(s):
Note from above that during that deploy, the following output appears:
[x.x.x.x] sudo: /home/cchq/www/monolith/releases/2022-09-24_18.49/python_env/bin/python manage.py migrate_multi --noinput
.
.
.
[x.x.x.x] out: This migration cannot be run, as it depends on code that has since been removed.
[x.x.x.x] out: To fix this, follow the instructions below to run this migration from a previous version of the code.
[x.x.x.x] out: In order to prevent this in the future, we recommend running migrations at least once every 6 weeks.
[x.x.x.x] out: For reference, the current code has not run migrations for 0 days.
[x.x.x.x] out:
[x.x.x.x] out: Run the following commands to run the historical migration and get up to date:
[x.x.x.x] out: With a cloud setup:
[x.x.x.x] out: commcare-cloud <env> fab setup_limited_release --set code_branch=116a6fc9556ac929e29a35fd2dfa673984820561 [240/1169]
[x.x.x.x] out:
[x.x.x.x] out: commcare-cloud <env> django-manage --release <release created by previous command> migrate_multi app_manager
[x.x.x.x] out:
[x.x.x.x] out: commcare-cloud <env> deploy commcare
I tried that route as follows:
commcare-cloud monolith fab setup_limited_release --set code_branch=116a6fc9556ac929e29a35fd2dfa673984820561
commcare-cloud monolith django-manage --release 2022-09-24_11.54 migrate_multi app_manager
commcare-cloud monolith deploy commcare
Note in step 2 the date/time release name is the output from step1
The first step finishes successfully with
Your private release is located here:
/home/cchq/www/monolith/releases/2022-09-24_19.35
It fails on the second step with:
(cchq) ccc@monolith:~/commcare-cloud$ commcare-cloud monolith django-manage --release 2022-09-24_11.54 migrate_multi app_manager
ssh ccc@x.x.x.x -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2022-09-[80/1551]
python_env/bin/python manage.py migrate_multi app_manager'"'"''
Ubuntu 18.04.6 LTS
commcare-cloud monolith deploy commcarebash: line 0: cd: /home/cchq/www/monolith/releases/2022-09-24_11.54: No such file or directory
bash: python_env/bin/python: No such file or directory
Connection to x.x.x.x closed.
(cchq) ccc@monolith:~/commcare-cloud$ commcare-cloud monolith django-manage --release 2022-09-24_19.35 migrate_multi app_manager
ssh ccc@x.x.x.x -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/releases/2022-09-24_19.35;
python_env/bin/python manage.py migrate_multi app_manager'"'"''
Ubuntu 18.04.6 LTS
Traceback (most recent call last):
File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
found = self.importer(used)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/__init__.py", line 1, in <module>
from .celery import app as celery_app # noqa
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/celery.py", line 22, in <module>
run_checks()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 416, in check
for pattern in self.url_patterns:
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 602, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
return import_module(self.urlconf_name)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/urls.py", line 7, in <module>
from corehq.extensions import extension_points
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/extensions/__init__.py", line 128, in <module>
from corehq.extensions.interface import CommCareExtensions, ResultFormat # noqa F401
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/extensions/interface.py", line 8, in <module>
from dimagi.utils.logging import notify_exception
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/ex-submodules/dimagi/utils/__init__.py", line 1, in <module>
from .decorators.profile import profile
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/ex-submodules/dimagi/utils/decorators/profile.py", line 15, in <module>
from corehq.util.decorators import ContextDecorator
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/util/decorators.py", line 10, in <module>
from celery.task import task
ModuleNotFoundError: No module named 'celery.task'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.9/logging/config.py", line 552, in configure
filters[name] = self.configure_filter(filters[name])
File "/usr/lib/python3.9/logging/config.py", line 689, in configure_filter
result = self.configure_custom(config)
File "/usr/lib/python3.9/logging/config.py", line 470, in configure_custom
c = self.resolve(c)
File "/usr/lib/python3.9/logging/config.py", line 398, in resolve
raise v
File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
found = self.importer(used)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/__init__.py", line 1, in <module>
from .celery import app as celery_app # noqa
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/celery.py", line 22, in <module>
run_checks()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 416, in check
for pattern in self.url_patterns:
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 602, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
return import_module(self.urlconf_name)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/urls.py", line 7, in <module>
from corehq.extensions import extension_points
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/extensions/__init__.py", line 128, in <module>
from corehq.extensions.interface import CommCareExtensions, ResultFormat # noqa F401
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/extensions/interface.py", line 8, in <module>
from dimagi.utils.logging import notify_exception
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/ex-submodules/dimagi/utils/__init__.py", line 1, in <module>
from .decorators.profile import profile
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/ex-submodules/dimagi/utils/decorators/profile.py", line 15, in <module>
from corehq.util.decorators import ContextDecorator
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/corehq/util/decorators.py", line 10, in <module>
from celery.task import task
ValueError: Cannot resolve 'corehq.util.log.HQRequestFilter': No module named 'celery.task'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/manage.py", line 208, in <module>
main()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/manage.py", line 43, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
django.setup()
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/manage.py", line 161, in _setup_once
_setup_once.setup(*args, **kw)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/__init__.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/home/cchq/www/monolith/releases/2022-09-24_19.35/python_env/lib/python3.9/site-packages/django/utils/log.py", line 75, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib/python3.9/logging/config.py", line 809, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python3.9/logging/config.py", line 554, in configure
raise ValueError('Unable to configure '
ValueError: Unable to configure filter 'hqrequest'
Connection to x.x.x.x closed.
Any assistance is appreciated!
EDIT just to add, I opened a venv in the new release directory and did a pip freeze to show active modules:
alembic==1.4.3
amqp==2.6.1
architect==0.5.6
asttokens==2.0.5
attrs==18.2.0
Babel==2.9.1
backcall==0.2.0
billiard==3.5.0.4
black==22.1.0
boto3==1.17.85
botocore==1.20.85
build==0.8.0
cachetools==5.0.0
celery @ https://github.com/dimagi/celery/raw/v4.1.1.2/releases/celery-4.1.1.2-py2.py3-none-any.whl
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==8.0.3
cloudant==2.14.0
CommcareTranslationChecker==0.9.7
contextlib2==0.6.0.post1
cryptography==3.4.8
csiphash==0.0.5
datadog==0.39.0
ddtrace==0.44.0
decorator==4.0.11
defusedxml==0.7.1
Deprecated==1.2.10
diff-match-patch==20200713
dimagi-memoized==1.1.3
Django==2.2.27
django-appconf==1.0.4
django-autoslug==1.9.8
django-braces==1.14.0
django-bulk-update==2.2.0
django-celery-results==1.0.4
django-compressor==2.4
django-countries==7.3.2
django-crispy-forms==1.10.0
django-cte==1.1.4
django-formtools==2.3
django-logentry-admin==1.0.6
django-oauth-toolkit==1.5.0
django-otp==0.9.4
django-phonenumber-field==5.2.0
django-prbac==1.0.1
django-recaptcha==2.0.6
django-redis==4.12.1
django-redis-sessions==0.6.1
django-statici18n==1.9.0
django-tastypie==0.14.3
django-transfer==0.4
django-two-factor-auth==1.13.1
django-user-agents==0.4.0
django-websocket-redis==0.6.0
djangorestframework==3.12.2
dnspython==1.15.0
dropbox==9.3.0
elasticsearch2==2.5.1
elasticsearch5==5.5.6
email-validator==1.1.3
et-xmlfile==1.0.1
ethiopian-date-converter==0.1.5
eulxml==1.1.3
executing==0.8.2
Faker==5.0.2
flower==0.9.2
gevent==21.8.0
ghdiff==0.4
google-api-core==2.5.0
google-api-python-client==2.32.0
google-auth==2.6.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
googleapis-common-protos==1.54.0
greenlet==1.1.2
gunicorn==20.0.4
hiredis==2.0.0
httpagentparser==1.9.0
httplib2==0.20.4
idna==2.10
importlib-metadata==4.11.3
intervaltree==3.1.0
ipython==8.0.1
iso8601==0.1.13
isodate==0.6.0
jdcal==1.4.1
jedi==0.18.1
Jinja2==2.11.3
jmespath==0.10.0
json-delta==2.0
jsonfield==2.1.1
jsonobject==0.9.10
jsonobject-couchdbkit==1.0.1
jsonpath-ng @ https://github.com/kaapstorm/python-jsonpath-rw/raw/wherenot+find_or_create/wheel/jsonpath_ng-1.5.2.2-py3-none-any.whl
jsonschema==3.2.0
jwcrypto==0.8
kafka-python==1.4.7
kombu==4.2.2.post1
laboratory==0.2.0
lxml==4.7.1
Mako==1.1.3
Markdown==3.3.6
MarkupSafe==1.1.1
matplotlib-inline==0.1.3
mypy-extensions==0.4.3
ndg-httpsclient==0.5.1
oauthlib==3.1.0
openpyxl==3.0.6
packaging==21.3
parso==0.8.3
pathspec==0.9.0
pep517==0.13.0
pexpect==4.8.0
phonenumberslite==8.12.10
pickle5==0.0.11
pickleshare==0.7.5
Pillow==9.0.1
pip-tools==6.8.0
platformdirs==2.4.1
ply==3.11
polib==1.1.0
prometheus-client==0.7.1
prompt-toolkit==3.0.26
protobuf==3.15.0
psycogreen==1.0.2
psycopg2==2.8.6
ptyprocess==0.7.0
pure-eval==0.2.2
py-KISSmetrics==1.1.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodome==3.10.1
PyGithub==1.54.1
Pygments==2.11.2
pygooglechart==0.4.0
PyJWT==1.7.1
pyOpenSSL==20.0.1
pyparsing==3.0.7
pyphonetics==0.5.3
pyrsistent==0.17.3
PySocks==1.7.1
python-dateutil==2.8.1
python-editor==1.0.4
python-imap==1.0.0
python-magic==0.4.22
python-mimeparse==1.6.0
python3-saml==1.12.0
pytz==2020.1
PyYAML==5.4.1
pyzxcvbn==0.8.0
qrcode==4.0.4
quickcache==0.5.4
rcssmin==1.0.6
redis==3.5.3
reportlab==3.5.55
requests==2.25.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rjsmin==1.1.0
rsa==4.8
s3transfer==0.4.2
schema==0.7.2
sentry-sdk==0.19.5
setproctitle==1.2.2
sh==1.14.2
simpleeval @ git+https://github.com/dimagi/simpleeval.git@d85c5a9f972c0f0416a1716bb06d1a3ebc83e7ec
simplejson==3.17.2
six==1.15.0
socketpool==0.5.3
sortedcontainers==2.3.0
sqlagg==0.17.2
SQLAlchemy==1.3.19
sqlparse==0.3.1
stack-data==0.1.4
stripe==2.54.0
suds-py3==1.4.4.1
tenacity==6.2.0
text-unidecode==1.3
tinys3==0.1.12
tomli==2.0.0
toposort==1.7
tornado==4.5.3
traitlets==5.1.1
tropo-webapi-python==0.1.3
turn-python==0.0.1
twilio==6.5.1
typing_extensions==4.1.1
ua-parser==0.10.0
Unidecode==1.2.0
uritemplate==4.1.1
urllib3==1.26.5
user-agents==2.2.0
uWSGI==2.0.19.1
vine==1.3.0
wcwidth==0.2.5
Werkzeug==1.0.1
wrapt==1.12.1
xlrd==2.0.1
xlwt==1.3.0
xmlsec==1.3.11
zipp==3.7.0
zope.event==4.5.0
zope.interface==5.4.0
The CC Cloud env has the following:
ansible==2.9.26
ansible-vault==1.1.1
args==0.1.0
attrs==21.2.0
bcrypt==3.2.0
boto3==1.18.57
botocore==1.21.57
certifi==2021.5.30
cffi==1.14.6
charset-normalizer==2.0.6
click==7.1.2
clint==0.5.1
-e git+https://github.com/dimagi/commcare-cloud.git@ed339d29a49e3f1fe3fed89072f916a0e1665c05#egg=commcare_cloud
couchdb-cluster-admin==0.7.2
cryptography==35.0.0
datadog==0.42.0
Deprecated==1.2.13
dimagi-memoized==1.1.3
dnspython==2.1.0
Fabric3==1.10.2.post3
gevent==21.8.0
greenlet==1.1.2
idna==2.6
importlib-metadata==3.1.0
importlib-resources==5.4.0
Jinja2==3.0.2
jinja2-cli==0.7.0
jmespath==0.10.0
jsonobject==2.0.0
MarkupSafe==2.0.1
netaddr==0.8.0
paramiko==2.7.2
passlib==1.7.4
pep517==0.10.0
pip-tools==6.4.0
pycparser==2.20
pycryptodome==3.10.4
PyGithub==1.55
PyJWT==2.4.0
PyNaCl==1.4.0
python-dateutil==2.8.2
pytz==2022.1
PyYAML==5.4.1
requests==2.26.0
s3transfer==0.5.0
simplejson==3.17.5
six==1.16.0
tabulate==0.8.9
toml==0.10.2
urllib3==1.26.7
wrapt==1.13.1
zipp==3.6.0
zope.event==4.5.0
zope.interface==5.4.0