Update-config failing

Thanks Simon.... there's definitely an issue with Redis (as well as celery and heartbeat):

EXCEPTION (Took   0.00s) redis          : Service check errored with exception 'TypeError("__init__() got an unexpected keyword argument 'health_check_interval'",)'
EXCEPTION (Took   0.00s) celery         : Service check errored with exception 'TypeError("__init__() got an unexpected keyword argument 'health_check_interval'",)'
EXCEPTION (Took   0.00s) heartbeat      : Service check errored with exception 'TypeError("__init__() got an unexpected keyword argument 'health_check_interval'",)'

The current Redis version is:
Redis server v=4.0.8 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64 build=78221d0efda3ae46

EDIT and this is the Redis log tail:

26071:M 05 May 11:20:54.534 * Running mode=standalone, port=6379.
26071:M 05 May 11:20:54.534 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
26071:M 05 May 11:20:54.534 # Server initialized
26071:M 05 May 11:21:02.097 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
26208:C 05 May 11:21:02.561 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26208:C 05 May 11:21:02.561 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=26208, just started
26208:C 05 May 11:21:02.561 # Configuration loaded
26209:M 05 May 11:21:02.564 * Running mode=standalone, port=6379.
26209:M 05 May 11:21:02.564 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
26209:M 05 May 11:21:02.565 # Server initialized

I'm not sure where that error is coming from - it would be helpful to have a full stacktrace. Can you run this from a Django shell:

from corehq.apps.hqadmin.service_checks import check_redis
check_redis()

You can get a shell either by using commcare-cloud:

cchq monolith django-manage shell

Or by directly logging to a machine and then running the following commands:

sudo -iu cchq
cd www/monolith/current
source python_env-3.6/bin/activate
python manage.py shell
1 Like

Thanks Simon - I'm about to do that.
Maybe not an issue but I do notice this after login when it loads the latest requirements and wondered if any of it is significant - note the no such job and version discrepancies:

Downloading dependencies from galaxy and pip
ansible-galaxy install -f -r /home/ccc/commcare-cloud/src/commcare_cloud/ansible/requirements.yml
/home/ccc
-bash: wait: %2: no such job
[WARNING]: - dependency andrewrothstein.couchdb (v2.1.4) (v2.1.4) from role
andrewrothstein.couchdb-cluster differs from already installed version
(v2.1.5), skipping
[WARNING]: - dependency ANXS.cron (None) from role tmpreaper differs from
already installed version (v1.0.2), skipping
[WARNING]: - dependency sansible.java (None) from role sansible.logstash
differs from already installed version (v2.1.4), skipping
[WARNING]: - dependency sansible.users_and_groups (None) from role
sansible.logstash differs from already installed version (v2.0.5), skipping
ansible-galaxy collection install -f -r /home/ccc/commcare-cloud/src/commcare_cloud/ansible/requirements.yml
To finish first-time installation, run manage-commcare-cloud configure
✓ origin already set to GitHub - dimagi/commcare-cloud: Tools for standing up and managing a CommCare HQ server environment
✓ /home/ccc/commcare-cloud/src/commcare_cloud/fab/config.py exists

It's possible an upgrade using apt was done on the server...

This is the output from the django-manage shell:

ssh 197.x.x.x -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env-3.6/bin/python manage.py shell'"'"''
Ubuntu 18.04.5 LTS
/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "manage.py", line 195, in <module>
    main()
  File "manage.py", line 41, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 308, in execute
    settings.INSTALLED_APPS
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settings.py", line 1437, in <module>
    SHARED_BLOB_DIR_NAME
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settingshelper.py", line 28, in __init__
    self.blob_dir = self._init_dir(blob_dir)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settingshelper.py", line 37, in _init_dir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/shared_monolith/blobdb'
Connection to 197.x.x.x closed.

And the second route:

(cchq) ccc@monolith:~/commcare-cloud$ sudo -iu cchq
(monolith) cchq@monolith:~$ cd www/monolith/current
(monolith) cchq@monolith:~/www/monolith/current$ source python_env-3.6/bin/activate
(python_env-3.6) (monolith) cchq@monolith:~/www/monolith/current$ python manage.py shell
/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "manage.py", line 195, in <module>
    main()
  File "manage.py", line 41, in main
    execute_from_command_line(sys.argv)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 308, in execute
    settings.INSTALLED_APPS
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settings.py", line 1437, in <module>
    SHARED_BLOB_DIR_NAME
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settingshelper.py", line 28, in __init__
    self.blob_dir = self._init_dir(blob_dir)
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/settingshelper.py", line 37, in _init_dir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/shared_monolith/blobdb'

With all the redis issues earlier, I tried redeploying Redis alone and get this Ansible error before applying:

TASK [DavidWittman.redis : add redis binaries to alternatives] ******************************************************************************************************************************
fatal: [197.211.237.144]: FAILED! => {"msg": "'dict object' has no attribute 'stdout_lines'"}

A quick update on this. For some reason redis seems to be behaving again and I was able to get further with a deploy. This time it failed and told me:

[197.x.x.x] out:                 A migration must be performed before this environment can be upgraded to the latest version
[197.x.x.x] out:                 of CommCareHQ. This migration is run using the management command populate_custom_data_fields.
[197.x.x.x] out:
[197.x.x.x] out:
[197.x.x.x] out:                 Run the following commands to run the migration and get up to date:
[197.x.x.x] out:
[197.x.x.x] out:                     commcare-cloud monolith deploy commcare --commcare-rev=bb82e5c3d2840d6e3e3a6f5ebf1a0c7e817f4613
[197.x.x.x] out:
[197.x.x.x] out:                     commcare-cloud monolith django-manage populate_custom_data_fields
[197.x.x.x] out:
[197.x.x.x] out:                     commcare-cloud monolith deploy commcare

So I am trying that right now. Possibly there was some other changelog I missed?
That bb82e deploy is running now but throwing this regularly:

[197.x.x.x] sudo: /home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/bin/python /home/cchq/www/monolith/releases/2021-05-06_10.41/manage.py preindex_everything --check
[197.x.x.x] out: 2021-05-06 10:48:14,171 INFO AXES: BEGIN LOG
[197.x.x.x] out: Traceback (most recent call last):
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/manage.py", line 189, in <module>
[197.x.x.x] out:     main()
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/manage.py", line 41, in main
[197.x.x.x] out:     execute_from_command_line(sys.argv)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
[197.x.x.x] out:     utility.execute()
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
[197.x.x.x] out:     self.fetch_command(subcommand).run_from_argv(self.argv)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
[197.x.x.x] out:     self.execute(*args, **cmd_options)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
[197.x.x.x] out:     output = self.handle(*args, **options)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/corehq/apps/hqadmin/management/commands/preindex_everything.py", line 59, in handle
[197.x.x.x] out:     if get_preindex_complete(head):
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/corehq/apps/hqadmin/management/commands/preindex_everything.py", line 138, in get_preindex_complete
[197.x.x.x] out:     return rcache.get(PREINDEX_COMPLETE_COMMIT, None) == head
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/cache.py", line 32, in _decorator
[197.x.x.x] out:     return method(self, *args, **kwargs)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/cache.py", line 81, in get
[197.x.x.x] out:     client=client)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/client/default.py", line 198, in get
[197.x.x.x] out:     client = self.get_client(write=False)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/client/default.py", line 94, in get_client
[197.x.x.x] out:     self._clients[index] = self.connect(index)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/client/default.py", line 107, in connect
[197.x.x.x] out:     return self.connection_factory.connect(self._server[index])
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/pool.py", line 65, in connect
[197.x.x.x] out:     connection = self.get_connection(params)
[197.x.x.x] out:   File "/home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/lib/python3.6/site-packages/django_redis/pool.py", line 77, in get_connection
[197.x.x.x] out:     return self.redis_client_cls(connection_pool=pool, **self.redis_client_cls_kwargs)
[197.x.x.x] out: TypeError: __init__() got an unexpected keyword argument 'health_check_interval'
[197.x.x.x] out:

It's eventually failing with:

[197.x.x.x] sudo: /home/cchq/www/monolith/releases/2021-05-06_10.41/python_env-3.6/bin/python manage.py send_email --to-admins --subject ' You can'"'"'t deploy to monolith yet. There'"'"'s a preindex in process.' 'Preindexing is taking a while, so hold tight and wait for an email saying it'"'"'s done. Thank you for using AWESOME DEPLOY.'

Is there something else I can do for it to recognize the 'health_check_interval' argument - perhaps something else was missed?
Thanks again!

Hi Ed, here goes:

FileExistsError: [Errno 17] File exists: '/opt/shared_monolith/blobdb'

Could this be a permissions error with that folder? You could try and run the deploy_shared_dir.yml playbook:

cchq monolith ap deploy_shared_dir.yml

unexpected keyword argument 'health_check_interval'

I finally tracked this down. This argument is passed to the Redis client from parameters in localsettings.py. To fix the issue you can upgrade the redis library in the current python venv:

sudo -iu cchq
cd www/monolith/current
source python_env-3.6/bin/activate
pip install -U django-redis==4.12.1

That should fix the issue with running commands in the current release.

Alternatively you could edit localsettings.py in the current release to remove that parameter from the Redis config.

populate_custom_data_fields migration

From time to time there are data migrations that take place which need to be run prior to updating the code beyond a certain point. This was one of those. I don't think a full deploy is necessary here and you should be able to create a new release from which to run the command:

commcare-cloud <env> fab setup_limited_release --set code_branch=bb82e5c3d2840d6e3e3a6f5ebf1a0c7e817f4613
commcare-cloud monolith django-manage --release <release name from previous command> populate_custom_data_fields
commcare-cloud monolith deploy commcare
1 Like

Thank you Simon! On the health_check_interval issue, I am getting this with pip install redis:

ERROR: redis-py-cluster 1.3.6 has requirement redis==2.10.6, but you'll have redis 3.5.3 which is incompatible.
ERROR: django-prbac 0.0.7 has requirement django<2.1,>=1.8, but you'll have django 3.2.2 which is incompatible.
Installing collected packages: typing-extensions, asgiref, sqlparse, Django, redis, django-redis
  Found existing installation: Django 1.11.26
    Uninstalling Django-1.11.26:
      Successfully uninstalled Django-1.11.26
  Found existing installation: redis 2.10.6
    Uninstalling redis-2.10.6:
      Successfully uninstalled redis-2.10.6
  Found existing installation: django-redis 4.10.0
    Uninstalling django-redis-4.10.0:
      Successfully uninstalled django-redis-4.10.0
Successfully installed Django-3.2.2 asgiref-3.3.4 django-redis-4.12.1 redis-3.5.3 sqlparse-0.4.1 typing-extensions-3.10.0.0

Is this an issue I should do something about?
In the mean time I'll disable the health_check_interval in localsettings.py
Thanks!

I'm not sure how much of an issue those depenedency mismatches are but probably safest to remove the param from the localsettings for now.

I managed to run the populate_custom_data_fields migration after editing the localsettings.py and then ran the deploy and a new issue arose during that deploy. The output is here:

Note:
[197.x.x.x] out: File "/home/cchq/www/monolith/releases/2021-05-06_13.26/python_env-3.6/lib/python3.6/site-packages/django/apps/config.py", line 178, in get_model
[197.x.x.x] out: return self.models[model_name.lower()]
[197.x.x.x] out: KeyError: 'apikeysettings'
LookupError: App 'hqwebapp' doesn't have a 'ApiKeySettings' model.

I appreciate your effort @Simon_Kelly !

What was the last commit in the release where you ran the migrations? I'm wondering if it was before this change was added: ensure apisettings have been migrated before deleting the model by snopoke · Pull Request #28561 · dimagi/commcare-hq · GitHub

If it does include that change then we might have to skip that migration but before doing that we should check if you have any of the data that's going to be migrated:

cchq monolith django-manage dbshell

> select * from hqwebapp_apikeysettings;
> select * from tastypie_apikey;
1 Like

If I'm understanding correctly, it was this comit: Merge pull request #28025 from dimagi/revert-27277-jls/migrate-Custom… · dimagi/commcare-hq@bb82e5c · GitHub so July 2, 2020 (from the post here)

Running the django-manage dbshell is throwing another issue:

ssh 197.x.x.x -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env-3.6/bin/python manage.py dbshell'"'"''
Ubuntu 18.04.5 LTS
/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "manage.py", line 195, in <module>
    main()
  File "manage.py", line 36, in main
    run_patches()
  File "manage.py", line 111, in run_patches
    patch_jsonfield()
  File "manage.py", line 126, in patch_jsonfield
    from jsonfield import JSONField
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/__init__.py", line 1, in <module>
    from .fields import JSONField, JSONCharField
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/fields.py", line 21, in <module>
    from .encoder import JSONEncoder
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/encoder.py", line 2, in <module>
    from django.utils import six, timezone
ImportError: cannot import name 'six'
Connection to 197.x.x.x closed.

I assume that's related to django.utils.six being absent

To work around that, I just ran sudo su - postgres and psql from there then connected to the commcarehq database and ran the queries. Here is the output:

commcarehq=# select * from hqwebapp_apikeysettings;
ERROR:  relation "hqwebapp_apikeysettings" does not exist
LINE 1: select * from hqwebapp_apikeysettings;

commcarehq=# select * from tastypie_apikey;
 id |                   key                    |            created            | user_id
----+------------------------------------------+-------------------------------+---------
  1 | *** | 2019-10-21 06:11:34.046931+00 |       1
  2 | *** | 2019-11-20 09:19:42.208624+00 |       5
  3 | *** | 2020-01-05 11:50:58.908474+00 |     648
  5 | *** | 2020-01-31 09:21:23.930751+00 |       7
  6 | *** | 2020-01-31 10:28:26.889869+00 |     390
  7 | *** | 2020-01-31 20:14:07.30836+00  |     661
  8 | *** | 2020-01-31 20:35:40.295587+00 |     662
  9 | *** | 2020-02-01 15:24:18.891733+00 |     666
 10 | *** | 2020-02-01 16:52:29.331956+00 |     667
 11 | *** | 2020-02-02 10:10:53.956809+00 |     280
 12 | *** | 2020-02-03 06:32:07.134061+00 |     663
 13 | *** | 2020-02-03 07:03:35.37959+00  |     630
 14 | *** | 2020-02-03 07:55:22.590552+00 |     629
 15 | *** | 2020-02-03 09:08:46.902905+00 |     647
 16 | *** | 2020-02-03 09:10:23.216115+00 |     637
 17 | *** | 2020-02-03 13:12:33.092785+00 |     632
 18 | *** | 2020-02-03 13:54:53.201333+00 |     638
 19 | *** | 2020-02-04 14:24:22.695895+00 |     665
 20 | *** | 2020-02-04 20:33:41.642875+00 |     683
 21 | *** | 2020-02-05 06:25:42.064984+00 |      12
 22 | *** | 2020-02-10 08:33:54.755361+00 |     633

Hi Ed, sorry I missed out SQL command, please also check this:

select count(*) from users_hqapikey k join tastypie_apikey t on k.key = t.key;

If that count is 0 then we'll need to migrate the data otherwise it's safe to skip that migration.

1 Like

Thanks @Simon_Kelly - it seems commcarehq.users_hqapikey is missing

commcarehq=# select count(*) from users_hqapikey k join tastypie_apikey t on k.key = t.key;
ERROR:  relation "users_hqapikey" does not exist
LINE 1: select count(*) from users_hqapikey k join tastypie_apikey t...

OK in that case let's fake the migration that's failing and we can migrate the data afterwards.

  1. Fake the migration:
python manage.py migrate_multi users 0016_hqapikey --fake
  1. Run the rest of the migrations:
python manage.py migrate_multi
  1. Migrate the API data
insert into users_hqapikey (key, created, user_id, role_id, domain, name, ip_allowlist)
select key, created, user_id, '', '', au.username, '{}'
from tastypie_apikey k join auth_user au on au.id = k.user_id;
1 Like

Which user should I be running this as? I've just tried running it directly as cchq

sudo -iu cchq
cd www/monolith/current
source python_env-3.6/bin/activate
python manage.py migrate_multi users 0016_hqapikey --fake

but we're still sitting with the issue of django.utils.six being missing:

(python_env-3.6) (monolith) cchq@monolith:~/www/monolith/current$ python manage.py migrate_multi users 0016_hqapikey --fake
/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "manage.py", line 195, in <module>
    main()
  File "manage.py", line 36, in main
    run_patches()
  File "manage.py", line 111, in run_patches
    patch_jsonfield()
  File "manage.py", line 126, in patch_jsonfield
    from jsonfield import JSONField
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/__init__.py", line 1, in <module>
    from .fields import JSONField, JSONCharField
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/fields.py", line 21, in <module>
    from .encoder import JSONEncoder
  File "/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/jsonfield/encoder.py", line 2, in <module>
    from django.utils import six, timezone
ImportError: cannot import name 'six'

I think this is because of the library upgrade we did earlier. Try this (with the virtual env activated):

pip install -U -r requirements/prod-requirements.txt

That should reinstall the dependencies at their correct versions.

1 Like

Thanks @Simon_Kelly that appeared to run fine and resolved the django.utils.six issue, the migrate_multi users 0016_hqapikey --fake threw an exception, the output is here:

... and of course users_hqapikey does not exist.

Thanks!

Hi Ed, it's odd that it couldn't find the migration. Were you running it from a new release?

You can use the showmigrations management command to list all available migrations.

As it turns out this is a bit more complex than I initially thought since that migration (0016_hqapikey) also includes the code to create the missing table. If we fake that migration we'll need to manually create the table afterwards:

$ python manage.py sqlmigrate users 0016_hqapikey

.... output will show the SQL that the migration would have run. You can copy this and run it directly in the DB shell.

$ python manage.py dbshell

> Paste the SQL from above command

Hey Simon, thanks for the response - here's the output from showmigrations (everything leading up to that included to confirm it's correct) - I'm not seeing 0016_hqapikey:

(cchq) ccc@monolith:~$ sudo -iu cchq
(monolith) cchq@monolith:~$ cd www/monolith/current
(monolith) cchq@monolith:~/www/monolith/current$ To finish first-time installation, run `manage-commcare-cloud configure`
source python_env-3.6/bin/activate
(python_env-3.6) (monolith) cchq@monolith:~/www/monolith/current$ python manage.py showmigrations
/home/cchq/www/monolith/releases/2020-01-13_09.37/python_env-3.6/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed f             rom release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-i             nstall-from-pypi>.
  """)
2021-05-10 09:57:13,693 INFO AXES: BEGIN LOG
aaa
 [X] 0001_squashed_0002_auto_20190213_2342
 [X] 0002_auto_20190214_2226
 [X] 0003_auto_20190215_1938_squashed_0004_auto_20190215_2103
 [X] 0004_auto_20190222_1955_squashed_0005_woman_name
 [X] 0005_auto_20190225_1822
 [X] 0006_auto_20190225_1900
 [X] 0007_auto_20190319_2225
 [X] 0008_denormalizedawc_denormalizedvillage
 [X] 0008_auto_20190410_1952
 [X] 0009_merge_20190411_1839
 [X] 0010_delete_womanhistory
accounting
 [X] 0001_squashed_0052_ensure_report_builder_plans
 [X] 0002_auto_20170222_2008
 [X] 0003_auto_20170328_2102
 [X] 0004_auto_20170404_0028
 [X] 0005_automatic_downgrade_adjustment_method
 [X] 0006_unique_active_domain_subscription
 [X] 0007_practice_mobile_workers
 [X] 0008_update_report_builder_included_feature_numbers
 [X] 0009_make_billingaccount_name_unique
 [X] 0010_remove_softwareproduct_product_type
 [X] 0011_remove_softwareproduct
 [X] 0012_replace__product_type__with__is_product
 [X] 0013_subscription_dates_check
 [X] 0014_paymentmethod__web_user__nonnullable
 [X] 0015_grandfather_login_as
 [X] 0016_grandfather_reportbuilder_5_pro
 [X] 0017_nonnullable_char_fields
 [X] 0018_alter_nonnullable_char_fields
 [X] 0019_standard_pricing_march_2018
 [X] 0020_payment_method__unique_together
 [X] 0021_standard_user_limit_march_2018
 [X] 0022_add__skip_auto_downgrade_reason
 [X] 0023_auto_20180501_1813
 [X] 0024_unique__transaction_id
 [X] 0025_auto_20180508_1952
 [X] 0026_auto_20180508_1956
 [X] 0027_auto_20180509_1857
 [X] 0028_auto_20180604_1757
 [X] 0029_auto_20180605_1826
 [X] 0030_softwareplan_max_domains
 [X] 0031_billingaccount_billing_admin_emails
 [X] 0032_customerinvoice_squashed_0036_customerbillingrecord
 [X] 0033_auto_20180709_1837
 [X] 0034_remove_subscription_date_delay_invoicing
 [X] 0032_billingaccount_invoicing_plan
 [X] 0034_merge_20180711_1828
 [X] 0035_merge_20180711_2039
 [X] 0036_domainuserhistory
 [X] 0035_enterprise_settings
 [X] 0037_merge_20180807_0915
 [X] 0038_remove_billingaccount_restrict_signup_email
 [X] 0039_auto_20180828_2258
 [X] 0040_auto_20181002_1721
 [X] 0041_auto_20190130_1709
 [X] 0042_domain_user_history__unique__and__nonnullable
 [X] 0043_grandfather_case_privs
 [X] 0044_grandfather_odata_privs
 [X] 0045_grandfather_data_forwarding_privs
 [X] 0046_new_plans
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
aggregate_ucrs
 [X] 0001_initial_squashed_0008_auto_20180625_1105 (8 squashed migrations)
 [X] 0002_auto_20180827_1148
app_manager
 [X] 0001_linked_app_domain
 [X] 0002_latestenabledbuildprofiles
 [X] 0003_auto_20190326_0853
 [X] 0004_latestenabledbuildprofiles_active
 [X] 0005_latestenabledbuildprofiles_domain
 [X] 0006_multi_master_linked_apps
 [X] 0007_add_linked_app_fields_to_es
 [X] 0008_remove_uses_master_app_form_ids
 [X] 0009_resourceoverride
 [X] 0010_run_add_resource_overrides
 [X] 0009_add_sqlglobalappconfig
 [X] 0010_merge_20191211_1921
 [X] 0011_merge_20191212_0936
 [X] 0010_sqlglobalappconfig
 [X] 0011_merge_20191212_0106
 [X] 0012_merge_20191217_0605
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
blobs
 [X] 0001_squashed_0009_domains
 [X] 0002_blobmeta
 [X] 0003_big_content
 [X] 0004_blobmeta_form_attachments
 [X] 0005_negative_id
 [X] 0006_restrict_form_attachments
 [X] 0007_drop_blobmeta_view
 [X] 0008_deletedblobmeta
 [X] 0009_delete_blobexpiration
 [X] 0010_auto_20191023_0938
captcha
 [X] 0001_initial
case_importer
 [X] 0001_initial
 [X] 0002_auto_20161206_1937
 [X] 0003_caseuploadrecord_couch_user_id
 [X] 0004_caseuploadrecord_case_type
 [X] 0005_caseuploadfilemeta
 [X] 0006_caseuploadrecord_upload_file_meta
 [X] 0007_auto_20161209_2004
 [X] 0008_caseuploadrecord_task_status_json
 [X] 0009_caseuploadrecord_comment
 [X] 0010_caseuploadformrecord
 [X] 0011_update_blob_paths
 [X] 0012_auto_20190405_1747
 [X] 0013_make_duplicates_errors
case_search
 [X] 0001_initial
 [X] 0002_auto_20161114_1901
 [X] 0003_casesearchqueryaddition
 [X] 0004_auto_20170518_2018
 [X] 0005_migrate_json_config
 [X] 0006_remove_casesearchconfig__config
 [X] 0007_auto_20170522_1506
 [X] 0008_auto_20180119_1716
cleanup
 [X] 0001_convert_change_feed_checkpoint_to_sql
 [X] 0002_convert_mc_checkpoint_to_sql
 [X] 0003_convert_fluff_checkpoints_to_sql
 [X] 0004_convert_ucr_checkpoints_to_sql
 [X] 0005_convert_mvp_checkpoints_to_sql
 [X] 0006_convert_report_es_checkpoints_to_sql
 [X] 0007_convert_es_checkpoints_to_sql
 [X] 0008_convert_sofabed_checkpoints_to_sql
 [X] 0009_convert_final_checkpoints_to_sql
 [X] 0010_rename_default_change_feed_checkpoint
 [X] 0011_merge_couch_sql_pillows
 [X] 0012_add_es_index_to_checkpoint_ids
 [X] 0013_migrate_kafka_checkpoint_format
commtrack
 [X] 0001_initial
 [X] 0002_stockstate_last_modified_form_id
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
couchforms
 [X] 0001_initial
 [X] 0002_auto_20170802_1505
 [X] 0003_auto_20170808_1212
 [X] 0004_unfinishedarchivestub
 [X] 0005_index_cleanup
 [X] 0006_unfinishedarchivestub_attempts
 [X] 0007_auto_20191210_2206
data_analytics
 [X] 0001_squashed_0004_auto_20150810_1710
 [X] 0002_maltrow_threshold
 [X] 0003_auto_20160205_0927
 [X] 0004_experienced_threshold
 [X] 0005_girrow
 [X] 0006_unique_girrow
 [X] 0007_auto_20160819_1423
 [X] 0008_auto_20161114_1903
 [X] 0009_remove_girrow_wam
data_dictionary
 [X] 0001_squashed_0002_auto_20161116_2209
 [X] 0002_auto_20161118_1537
 [X] 0003_auto_20161128_2047
 [X] 0004_auto_20161130_2125
 [X] 0005_casetype_fully_generated
 [X] 0006_caseproperty_group
 [X] 0007_property_type_choices
data_interfaces
 [X] 0001_initial
 [X] 0002_remove_exists_option
 [X] 0003_update__automaticupdaterulecriteria__match_type__choices
 [X] 0004_optional_modified_date_and_prop_type_choices
 [X] 0005_remove_match_type_choices
 [X] 0006_case_rule_refactor
 [X] 0007_logging_models
 [X] 0008_update_case_rulesubmission
 [X] 0009_scheduling_integration
 [X] 0010_automaticupdaterule_workflow
 [X] 0011_domaincaserulerun_num_creates
 [X] 0012_createscheduleinstanceactiondefinition_reset_case_property_name
 [X] 0013_createscheduleinstanceactiondefinition_scheduler_module_info
 [X] 0014_createscheduleinstanceactiondefinition_start_date_case_property
 [X] 0015_automaticupdaterule_locked_for_editing
 [X] 0016_createscheduleinstanceactiondefinition_specific_start_date
 [X] 0017_alter_domaincaserulerun
 [X] 0018_check_for_rule_migration
 [X] 0019_remove_old_rule_models
 [X] 0020_make_migrated_nullable
 [X] 0021_remove_automaticupdaterule_migrated
 [X] 0022_domaincaserulerun_case_type
dhis2
 [X] 0001_initial
 [X] 0002_auto_20170322_1323
 [X] 0003_jsonapilog_log_level
 [X] 0004_auto_20171122_0827
 [X] 0005_delete_jsonapilog
django_celery_results
 [X] 0001_initial
 [X] 0002_add_task_name_args_kwargs
 [X] 0003_auto_20181106_1101
django_digest
 [X] 0001_initial
django_prbac
 [X] 0001_initial
doctypemigrations
 [X] 0001_initial
 [X] 0002_auto_20150924_1930
 [X] 0003_doctypemigration_cleanup_complete
 [X] 0004_auto_20151001_1809
 [X] 0005_auto_20151013_0819
 [X] 0006_domain_migration_20151118
 [X] 0007_apps_20160129
domain
 [X] 0001_initial
 [X] 0002_auto_20171020_1428
 [X] 0003_auto_20180525_1551
 [X] 0004_domainauditrecordentry
domain_migration_flags
 [X] 0001_initial
 [X] 0002_migrate_data_from_tzmigration
 [X] 0003_add_migration_dates
dropbox
 [X] 0001_initial
export
 [X] 0001_initial
 [X] 0002_datafile
 [X] 0003_emailexportwhendonerequest
 [X] 0004_datafile_delete_after
 [X] 0005_datafile_blobmeta
 [X] 0006_delete_dailysavedexportnotification
 [X] 0007_auto_20190906_0149
 [X] 0008_auto_20190906_2008
fixtures
 [X] 0001_initial
 [X] 0002_rm_blobdb_domain_fixtures
 [X] 0003_rm_blobdb_domain_fixtures
form_processor
 [X] 0001_initial
 [X] 0002_xformattachmentsql
 [X] 0003_auto_20151104_2226
 [X] 0004_create_commcarecasesql
 [X] 0005_make_case_uuid_unique_indexed
 [X] 0006_commcarecaseindexsql
 [X] 0007_index_case_uuid_on_commcarecaseindex
 [X] 0008_add_index_for_caseforms_case_uuid
 [X] 0009_add_xform_operation_model_and_state
 [X] 0010_add_auth_and_openrosa_fields
 [X] 0011_add_fields_for_deprecation
 [X] 0012_xforminstancesql_problem
 [X] 0013_caseattachmentsql
 [X] 0014_caseattachmentsql_index_on_foreign_key
 [X] 0015_change_related_names_for_form_attachment
 [X] 0016_index_case_attachment_uuid
 [X] 0017_problem_to_text_field
 [X] 0018_xforminstancesql_user_id
 [X] 0019_allow_closed_by_null
 [X] 0020_rename_index_relationship
 [X] 0021_change_case_forms_related_name
 [X] 0022_set_default_value_for_case_json
 [X] 0023_make_case_name_top_level
 [X] 0024_rename_case_type
 [X] 0025_caseforms_server_date
 [X] 0026_caseforms_to_casetransaction
 [X] 0027_allow_null_form_uuid_in_case_transaction
 [X] 0025_add_dict_defaults_for_xform
 [X] 0026_xforminstancesql_initial_processing_complete
 [X] 0028_merge
 [X] 0029_drop_not_null_from_opened_on_by
 [X] 0030_casetransaction_revoked
 [X] 0031_add_details_field_to_case_transaction
 [X] 0032_change_transaction_related_name
 [X] 0033_commcarecasesql_location_uuid
 [X] 0034_location_id_index
 [X] 0035_remove_varchar_pattern_ops_indexes
 [X] 0036_cleanup_models
 [X] 0037_get_form_by_id_fn
 [X] 0038_form_functions
 [X] 0039_auto_20151130_1748
 [X] 0039_case_functions
 [X] 0040_save_functions
 [X] 0041_noop_specify_table_names
 [X] 0042_change_choice_values
 [X] 0043_rename_to_match
 [X] 0044_remove_foreign_key_to_case
 [X] 0045_casetransaction_sync_log_id
 [X] 0046_add_not_null_constraint_to_owner_id
 [X] 0047_add_deleted_state
 [X] 0048_attachment_content_length_blob_id
 [X] 0049_case_attachment_props
 [X] 0050_allow_null_content_type
 [X] 0051_auto_20160224_0922
 [X] 0052_auto_20160224_1011
 [X] 0053_add_deletion_fields
 [X] 0054_ledgertransaction
 [X] 0055_daily_consumption
 [X] 0056_last_modified_form_id
 [X] 0057_ledger_value_domain_location
 [X] 0058_update_indexes
 [X] 0059_remove_ledgervalue_location_id
 [X] 0060_convert_case_ids_to_foreign_keys
 [X] 0061_blob_bucket
 [X] 0062_auto_20160905_0938
 [X] 0063_auto_20160908_0954
 [X] 0064_auto_20170725_0943
 [X] 0065_auto_20170725_1339
 [X] 0066_auto_20170818_2020
 [X] 0067_auto_20170915_1506
 [X] 0068_xforminstancesql_modified_on
 [X] 0069_add_metafields
 [X] 0070_auto_20180329_1016
 [X] 0071_case_attachment_refactor
 [X] 0072_case_attachment_drops
 [X] 0073_drop_case_uuid_like_index
 [X] 0074_casetransaction__client_date
 [X] 0075_auto_20181026_0951
 [X] 0076_form_attachment_fk
 [X] 0077_null_properties
 [X] 0078_blobmeta_migrated_check
 [X] 0079_add_xmlns_index
 [X] 0080_migrate_delta_1_add_column
 [X] 0081_migrate_delta_2_create_trigger
 [X] 0082_migrate_delta_3_backfill_notnull
 [X] 0083_migrate_delta_4_switch_columns
 [X] 0084_migrate_delta_5_alter_field
 [X] 0085_migrate_balance_1_add_column
 [X] 0086_migrate_balance_2_create_trigger
 [X] 0087_migrate_balance_3_backfill_notnull
 [X] 0088_migrate_balance_4_switch_columns
 [X] 0089_migrate_balance_5_alter_field
 [X] 0090_auto_20190523_0833
 [X] 0091_auto_20190603_2023
hqadmin
 [X] 0001_initial
 [X] 0002_vcmmigrationaudit
 [X] 0003_auto_20160715_1543
 [X] 0004_auto_20160715_1547
 [X] 0005_auto_20160715_1612
 [X] 0006_esrestorepillowcheckpoints
 [X] 0007_esrestorepillowcheckpoint_datefield
 [X] 0008_delete_vcmmigration
 [X] 0009_auto_20170315_1322
hqwebapp
 [X] 0001_initial
 [X] 0002_auto_20171121_1803
 [X] 0003_maintenancealert_domains
icds
 [X] 0001_initial
 [X] 0002_auto_20190520_1215
 [X] 0003_auto_20190520_1859
 [X] 0004_auto_20190524_1657
 [X] 0005_hostedccz_note
 [X] 0006_hostedccz_status
icds_reports
 [X] 0001_squashed_0110_aww_incentive_report_monthly (116 squashed migrations)
 [X] 0111_service_delivery_dashboard
 [X] 0112_aggawcdailyview_agglsmonthly_awwincentivereportmonthly_childhealthmonthlyview_servicedeliverymonthly
 [X] 0113_service_delivery_dashboard
 [X] 0114_local_tables
 [X] 0115_citusdashboarddiff_citusdashboardexception_citusdashboardtiming
 [X] 0116_daily_attendance_index
 [X] 0117_awc_location_local
 [X] 0118_monthly_pk
 [X] 0119_distribute_tables
 [X] 0120_auto_20190529_1459
 [X] 0121_auto_20190530_2136
 [X] 0122_incentive_columns
 [X] 0123_weighing_new_columng_agg_awc
 [X] 0124_update_agg_awc_view
 [X] 0125_aggregatethrform
 [X] 0126_agg_awc_image_column
 [X] 0127_thr_report_view
 [X] 0128_nic_indicators
 [X] 0129_aggregationrecord
 [X] 0130_thr_mother_column_agg_awc
 [X] 0131_update_views_For_thr_colums
 [X] 0132_update_nic_view
 [X] 0133_auto_20191001_0835
 [X] 0134_audit_entry_record_response_code
 [X] 0135_index_gm_forms_month
 [X] 0136_add_infra_field
 [X] 0137_pse_columns_awc_daily
 [X] 0138_add_db_index_inactive_aww
 [X] 0139_update_thr_view (1 squashed migrations)
 [X] 0140_auto_20191024_1048
 [X] 0141_remove_pse_eligible_daily
 [X] 0142_gm_column_agg_awc
 [X] 0143_remove_group_by_thr_view
 [X] 0143_add_ucrreconciliationstatus_squashed
 [X] 0144_merge_20191106_1522
 [X] 0145_add_interval_to_aggregationrecord
 [X] 0146_auto_20191030_1120
 [X] 0147_mwcd_report_view
 [X] 0148_auto_20191112_2123
 [X] 0149_auto_20191115_1142
 [X] 0150_update_agg_awc_monthly_view
 [X] 0151_fix_sdd_view
 [X] 0152_dashboarduseractivityreport
 [X] 0153_daily_attendance_state
 [X] 0154_child_health_state
 [X] 0155_added_new_fields_css_record_monthly_and_delivery_forms
 [X] 0156_auto_20191217_0932
 [X] 0157_updated_unique_constraint
 [X] 0158_add_in_month_column
 [X] 0159_update_agg_awc_monthly_view_visit_fields
ilsgateway
 [X] 0001_initial
 [X] 0002_auto_20160104_1600
 [X] 0002_auto_20151204_2142
 [X] 0003_merge
 [X] 0003_auto_20160104_1012
 [X] 0004_merge
 [X] 0005_add_pending_reporting_data_recalculation
 [X] 0006_delete_supply_point_warehouse_record
 [X] 0007_auto_20160322_1434
 [X] 0008_auto_20160401_0807
 [X] 0009_auto_20160413_1311
 [X] 0010_auto_20160830_1923
 [X] 0011_auto_20170714_1411
integration
 [X] 0001_initial
ivr
 [X] 0001_initial
 [X] 0002_call_app_id
linked_domain
 [X] 0001_initial
 [X] 0002_migrate_linked_apps
 [X] 0003_auto_20180207_0857
 [X] 0004_domainlinkhistory
 [X] 0005_migrate_linked_app_toggle
 [X] 0006_auto_20180208_1139
 [X] 0007_auto_20180215_1434
 [X] 0008_hidden_history
 [X] 0008_auto_20190405_1747
 [X] 0009_merge_20190408_1729
locations
 [X] 0001_initial
 [X] 0002_auto_20160420_2105
 [X] 0003_remove_null_True
 [X] 0004_auto_20160914_2030
 [X] 0005_locationtype_include_without_expanding
 [X] 0006_locationfixtureconfiguration
 [X] 0007_add_blank_true
 [X] 0008_increase_name_max_length
 [X] 0009_create_location_index
 [X] 0009_add_location_user_fields
 [X] 0010_merge_20170511_0724
 [X] 0011_locationtype_include_only
 [X] 0012_get_location_fixture_ids
 [X] 0013_rm_mptt_fields
 [X] 0013_adding_indices_for_warehouse
 [X] 0014_merge_20180601_0608
 [X] 0015_create_related_locations
 [X] 0016_locationrelation_distance
 [X] 0017_locationrelation_last_modified
m4change
 [X] 0001_initial
motech
 [X] 0001_initial
 [X] 0002_requestlog_payload_id
 [X] 0003_auto_20200102_1006
notifications
 [X] 0001_squashed_0003_auto_20160504_2049 (3 squashed migrations)
 [X] 0002_auto_20160505_2058
 [X] 0003_lastseennotification
 [X] 0004_auto_20160830_2002
 [X] 0005_domain_specific_notifications
 [X] 0006_add_billing_type
 [X] 0007_auto_20170704_1643
ota
 [X] 0001_initial
 [X] 0002_alter_db_index
 [X] 0003_add_serial_id_model
 [X] 0004_mobilerecoverymeasure
 [X] 0005_rm_max_length_from_integer
 [X] 0006_one_reinstall_measure
 [X] 0007_update_blob_paths
 [X] 0008_auto_20190108_1808
 [X] 0009_devicelogrequest
otp_static
 [X] 0001_initial
otp_totp
 [X] 0001_initial
phone
 [X] 0001_initial
 [X] 0002_synclogsql
 [X] 0003_auto_20190405_1752
 [X] 0004_auto_20191021_1308
phonelog
 [X] 0001_initial
 [X] 0002_auto_20160219_0951
 [X] 0003_auto_20160225_0420
 [X] 0002_auto_20160218_1613
 [X] 0004_merge
 [X] 0005_add_forceclose_entry_20160408_1530
 [X] 0006_usererrorentry_context_node
 [X] 0007_devicelog_indexes
 [X] 0008_devicelog_varchar_index
 [X] 0009_userentry_server_date
 [X] 0010_rename_device_model
 [X] 0011_partition_devicelogentry
 [X] 0012_server_date_not_null
 [X] 0013_delete_olddevicereportentry
 [X] 0014_auto_20170718_2039
pillow_retry
 [X] 0001_initial
 [X] 0002_pillowerror_queued
 [X] 0003_auto_20151002_0944
 [X] 0004_auto_drop_legacy_ucr_errors
 [X] 0005_auto_20160923_1913
 [X] 0006_auto_20170615_0327
 [X] 0007_remove_pillowerror_queued
 [X] 0008_index_cleanup
pillowtop
 [X] 0001_initial
 [X] 0002_djangopillowcheckpoint_sequence_format
 [X] 0003_auto_20170411_1957
 [X] 0004_offset_to_big_int
 [X] 0005_kafkacheckpoint_doc_modification_time
products
 [X] 0001_initial
project_limits
 [X] 0001_initial
reminders
 [X] 0001_initial
repeaters
 [X] 0001_adjust_auth_field_format
reports
 [X] 0001_initial
 [X] 0002_auto_20171121_1803
 [X] 0003_multiselect_report_filters_are_lists
saved_reports
 [X] 0001_initial
scheduling
 [X] 0001_initial
 [X] 0002_add__custom_content__is_test
 [X] 0003_add_custom_metadata
 [X] 0004_timedschedule_start_offset
 [X] 0005_timedschedule_start_day_of_week
 [X] 0006_add_broadcast_recipients
 [X] 0007_add_schedule_ui_type
 [X] 0008_timedschedule_event_type
 [X] 0009_randomtimedevent
 [X] 0010_casepropertytimedevent
 [X] 0011_add_broadcast_deleted_flag
 [X] 0012_add_schedule_deleted_flag
 [X] 0013_advanced_survey_options
 [X] 0014_remove_is_test
 [X] 0015_rename_schedule_length_to_repeat_every
 [X] 0016_location_type_filter
 [X] 0017_update_ui_type
 [X] 0018_minutes_to_wait
 [X] 0019_add_use_utc_as_default_timezone
 [X] 0020_migratedreminder
 [X] 0021_add_user_data_filter
 [X] 0022_add_stop_date_case_property_name
 [X] 0023_add_remaining_content_models
 [X] 0024_app_id
scheduling_partitioned
 [X] 0001_initial
 [X] 0002_case_schedule_instances
 [X] 0003_add_last_reset_case_property_value
 [X] 0004_case_schedule_instance_unique_together
 [X] 0005_timed_schedule_instance_schedule_revision
 [X] 0006_unique_indexes
 [X] 0007_index_cleanup
sessions
 [X] 0001_initial
sites
 [X] 0001_initial
 [X] 0002_alter_domain_unique
sms
 [X] 0001_initial
 [X] 0002_add_selfregistrationinvitation
 [X] 0003_add_backend_models
 [X] 0004_add_sqlivrbackend_sqlkookoobackend
 [X] 0005_remove_mobilebackend_unique_constraint
 [X] 0006_add_migrationstatus
 [X] 0007_check_for_backend_migration
 [X] 0008_add_backend_mapping_unique_constraint
 [X] 0009_check_for_domain_default_backend_migration
 [X] 0010_update_sqlmobilebackend_couch_id
 [X] 0011_alter_inbound_api_key
 [X] 0012_add_lastreadmessage_expectedcallback
 [X] 0013_check_for_log_migration
 [X] 0014_add_queuedsms
 [X] 0015_rename_phonenumber_to_phoneblacklist
 [X] 0016_add_phonenumber
 [X] 0017_update_phoneblacklist
 [X] 0018_check_for_phone_number_migration
 [X] 0019_add_new_registration_invitation_fields
 [X] 0020_remove_selfregistrationinvitation_odk_url
 [X] 0021_add_keyword
 [X] 0022_keyword_last_modified
 [X] 0023_check_for_keyword_migration
 [X] 0024_add_is_two_way
 [X] 0025_add_custom_metadata
 [X] 0026_add_messagingsubevent_case_id_index
 [X] 0027_ivorycoastmtnbackend
 [X] 0028_messagingevent_source
 [X] 0029_daily_outbound_sms_limit_reached
 [X] 0030_karixbackend
 [X] 0031_airteltclbackend
 [X] 0032_couch_id_blank
 [X] 0033_starfishbackend
 [X] 0034_auto_20191007_0756_noop
 [X] 0036_index_cleanup
 [X] 0037_app_id
smsbillables
 [X] 0001_initial
 [X] 0002_bootstrap
 [X] 0003_update_twilio_rates_outgoing
 [X] 0004_bootstrap_smsgh_rates
 [X] 0005_update_http_backend_criteria
 [X] 0006_remove_smsbillable_api_response
 [X] 0007_smsbillable_multipart_count
 [X] 0008__multipart_count__non_nullable
 [X] 0009_smsbillable_direct_gateway_fee
 [X] 0010_gateway_fee_amount_null
 [X] 0011_date_to_datetime
 [X] 0012_remove_max_length
 [X] 0013_auto_20160826_1531
 [X] 0014_bootstrap_apposit_rates
 [X] 0015_bootstrap_icds_rates
 [X] 0016_smsgatewayfeecriteria_is_active
 [X] 0017_deactivate_grapevine_instance_fee_criteria
 [X] 0018_index__date_sent
 [X] 0019_deactivate_hardcoded_twilio_gateway_fees
 [X] 0020_update_sislog_fees
smsforms
 [X] 0001_initial
 [X] 0002_add_state_tracking_fields
 [X] 0003_make_fields_not_null
sql_accessors
 [X] 0001_initial
 [X] 0002_add_sync_functions
 [X] 0003_get_forms_by_user_id_functions
 [X] 0004_get_modified_since_functions
 [X] 0005_add_fields_to_attachments
 [X] 0006_add_fields_to_case_attachments
 [X] 0007_rename_get_case_attachment_by_name
 [X] 0008_get_case_by_external_id
 [X] 0009_ledger_accessors
 [X] 0010_update_state_type_values
 [X] 0011_get_case_types_for_domain
 [X] 0012_soft_deletion
 [X] 0012_get_case_transactions_by_type
 [X] 0013_merge
 [X] 0014_ledger_transactions
 [X] 0015_get_form_ids_in_domain
 [X] 0016_ledger_values_by_product_ids
 [X] 0015_ledger_transactions_update
 [X] 0017_merge
 [X] 0018_save_ledger_values_rebuild
 [X] 0019_get_case_ids_in_domain_by_owners
 [X] 0020_soft_delete_cases
 [X] 0021_get_ledger_values_for_cases
 [X] 0022_last_modified_form_id
 [X] 0023_update_get_case_form_ids
 [X] 0024_delete_ledgers
 [X] 0024_update_save_ledger_values
 [X] 0025_update_get_ledger_values_for_cases
 [X] 0026_get_all_ledger_values_since
 [X] 0027_audit_fixups
 [X] 0028_rename_get_multiple_forms_attachments
 [X] 0029_get_case_ids_in_domain
 [X] 0030_index_changes
 [X] 0031_remove_get_ledger_values_for_product_ids
 [X] 0032_exclude_deleted_cases
 [X] 0033_remove_location_id_from_ledgers
 [X] 0034_update_reindex_functions
 [X] 0035_add_undelete_functions
 [X] 0036_exclude_deleted
 [X] 0037_delete_ledgers_with_case
 [X] 0038_attachment_bucket
 [X] 0038_save_case_deletion_fields
 [X] 0039_update_reindex_queries
 [X] 0040_fix_case_reindex_query
 [X] 0041_latest_ledger_transactions_fix
 [X] 0042_write_blob_bucket
 [X] 0043_faster_get_reverse_indexed_cases
 [X] 0044_remove_get_case_types_for_domain
 [X] 0045_drop_case_modified_since
 [X] 0046_livequery_sql
 [X] 0047_livequery_sql
 [X] 0048_livequery_sql
 [X] 0049_remove_save_case
 [X] 0050_remove_get_extension_case_ids
 [X] 0050_faster_get_reverse_indexed_cases
 [X] 0051_merge_20170724_1255
 [X] 0052_save_ledgers_fix
 [X] 0053_drop_unused_sql
 [X] 0054_drop_reindexa_accessor_functions
 [X] 0055_set_form_modified_on
 [X] 0056_add_hashlib_functions
 [X] 0057_filter_get_reverse_indexed_cases
 [X] 0058_drop_old_reverse_indexed_fn
 [X] 0059_case_attachment_refactor
 [X] 0060_case_attachment_drops
 [X] 0061_check_transaction_sort_order
 [X] 0062_form_attachments
 [X] 0063_get_ledger_values_for_cases_2
sql_proxy_accessors
 [X] 0001_initial
 [X] 0002_add_sync_functions
 [X] 0003_get_forms_by_user_id_functions
 [X] 0004_get_modified_since_functions
 [X] 0005_rename_get_case_attachment_by_name
 [X] 0006_get_case_by_external_id
 [X] 0007_ledger_accessors
 [X] 0008_get_case_types_for_domain
 [X] 0009_soft_delete
 [X] 0009_get_transactions_by_type
 [X] 0010_merge
 [X] 0011_ledger_transactions
 [X] 0012_get_form_ids_in_domain
 [X] 0013_ledger_values_by_product_ids
 [X] 0012_ledger_transactions_update
 [X] 0014_merge
 [X] 0015_save_ledger_values_rebuild
 [X] 0016_get_case_ids_in_domain_by_owners
 [X] 0017_soft_delete_cases
 [X] 0018_get_ledger_values_for_cases
 [X] 0019_delete_ledgers
 [X] 0020_update_get_ledger_values_for_cases
 [X] 0021_get_all_ledger_values_modified_since
 [X] 0022_audit_fixups
 [X] 0023_rename_get_multiple_forms_attachments
 [X] 0024_get_case_ids_in_domain
 [X] 0025_index_changes
 [X] 0026_remove_get_ledger_values_for_product_ids
 [X] 0027_add_undelete_functions
 [X] 0030_exclude_deleted
 [X] 0031_write_blob_bucket
 [X] 0032_remove_get_cases_by_domain
 [X] 0033_drop_case_modified_since
 [X] 0034_livequery_sql
 [X] 0035_livequery_sql
 [X] 0036_remove_save_case
 [X] 0037_remove_get_extension_case_ids
 [X] 0038_drop_unused_sql
 [X] 0039_filter_get_reverse_indexed_cases
 [X] 0040_drop_old_reverse_indexed_fn
 [X] 0041_case_attachment_refactor
 [X] 0042_case_attachment_drops
 [X] 0043_blobmeta
 [X] 0044_blobmeta_form_attachments
 [X] 0045_form_attachments
 [X] 0046_get_ledger_values_for_cases_2
sql_proxy_standby_accessors
 [X] 0001_initial
start_enterprise
 [X] 0001_initial
stock
 [X] 0001_squashed_0002_stockreport_server_date
tastypie
 [X] 0001_initial
 [X] 0002_api_access_url_length
telerivet
 [X] 0001_initial
 [X] 0002_add_index_on_webhook_secret
toggle_ui
 [X] 0001_reconcile_commtrack_flags
translations
 [X] 0001_initial
 [X] 0002_transifexblacklist
 [X] 0003_auto_20190318_1850
 [X] 0004_auto_20190318_1905
 [X] 0005_auto_20190405_1747
two_factor
 [X] 0001_initial
 [X] 0002_auto_20150110_0810
 [X] 0003_auto_20150817_1733
 [X] 0004_auto_20160205_1827
 [X] 0005_auto_20160224_0450
tzmigration
 [X] 0001_initial
 [X] 0002_delete_timezonemigrationprogress
userreports
 [X] 0001_initial
 [X] 0002_migrate_translation_transforms
 [X] 0003_auto_20170321_2019
 [X] 0004_auto_20170330_2049
 [X] 0005_asyncindicator_unsuccessful_attempts
 [X] 0006_add_index_to_domain
 [X] 0007_index_indicator_config_ids
 [X] 0008_new_table_name_views
 [X] 0009_rename_ucr_tables
 [X] 0010_add_invaliducrdata
 [X] 0011_datasourceactionlog
 [X] 0012_datasourceactionlog_skip_destructive
 [X] 0013_reportcomparisondiff_reportcomparisonexception_reportcomparisontiming
 [X] 0014_auto_20190529_1458
 [X] 0015_auto_20190701_2006
 [X] 0016_change_primary_key_to_bigint
 [X] 0017_index_cleanup
users
 [X] 0001_add_location_permission
 [X] 0002_domainrequest
 [X] 0003_roles_permissions_update
 [X] 0004_rm_role_id_from_admins
 [X] 0005_add_user_reporting_metadata
 [X] 0006_auto_20191225_1207
util
 [X] 0001_initial
zapier
 [X] 0001_initial
 [X] 0002_auto_20170117_1756
zipline
 [X] 0001_initial
 [X] 0002_add_package_and_product_models

Hi Ed. You'll need to run the commands from an updated release. Were you able to create a new release successfully?

You can use cchq monolith fab setup_limited_release to create a new release folder.

Once that's done you can use it to run the commands.