Error during migrate - HQ Update on monolith

I'm having issues upgrading HQ on one of our servers. I've run through the changelog up to 2022-08-11 historical-auditcare-data-migration and have opted to check out Commcare Cloud at ed339d29a49e3f1fe3fed89072f916a0e1665c05 and deploy HQ at 694338d34ec552ded83703160344517b4892cf08

Any help is greatly appreciated!

It stops right after

[] sudo: /home/cchq/www/monolith/releases/2022-09-13_10.52/python_env/bin/python migrate_multi --noinput
[] out: 2022-09-13 11:02:40,240 INFO [dimagi.utils.couch.sync_docs] synced 'repeaters' in couchdb`
[] out: Traceback (most recent call last):
[] out:   File "src/gevent/", line 906, in
[] out:   File "/home/cchq/www/monolith/releases/2022-09-13_10.52/corehq/sql_db/management/commands/", line 41, in migrate_db
[] out:     call_command('migrate', *args, **call_options)
[] out: AttributeError: 'NoneType' object has no attribute 'enabled'
[] out: 2022-09-13T11:02:40Z <Greenlet at 0x7f3b222ab9d0: migrate_db('default')> failed with AttributeError

The full output is below:
cchq -

Hi Ed

That does look like a bug, we'll look at it and get back to you.

1 Like

Hi Ed,

The issue is that the feature flag this migration references has since been deleted. I created this branch bugfix/linked-domains-migration off of the commit hash you referenced 694338d34ec552ded83703160344517b4892cf08, and the branch includes an additional commit that should resolve the issue you encountered. You can deploy from this branch using the --commcare-rev option (e.g., cchq <env> deploy --commcare-rev bugfix/linked-domains-migration). I hope that helps and reach out if you run into any issues.

1 Like

Very helpful, thank you Graham. I'll likely be able to provide feedback before tomorrow lunch time.

Hi all, it seems the migration went well but I've since been unable to deploy. It seems no matter which release I deploy, I get the following error consistently - starting at the end of the test phase:

[] Executing task 'record_successful_release'
[] sudo: echo '/home/cchq/www/monolith/releases/2022-09-14_09.50' >> "$(echo RELEASES.txt)"

Disconnecting from done.
>> Sending email: CommCare HQ deploy successful - monolith
Ubuntu 18.04.6 LTS
Connection to closed.
commcare-cloud monolith django-manage record_deploy_success --user ccc --environment monolith --url ' det$
cted dubious ownership in repository at '"'"'/home/cchq/www/monolith/releases/2022-09-14_08.59'"'"'
To add an exception for this directory, call:

        git config --global --add /home/cchq/www/monolith/releases/2022-09-14_08.59...00549f6cebdb3a86d9222bd8d5db42495d76b4dd' --minutes 13 -$
commit 00549f6cebdb3a86d9222bd8d5db42495d76b4dd
ssh ccc@ -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_$
nv/bin/python record_deploy_success --user ccc --environment monolith --url '"'"'"'"'"'"'"'"' de
tected dubious ownership in repository at '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'/home/cchq/www/monolith/releases/2022-09-14_08.59'"'"'"'"'"'"'"
To add an exception for this directory, call:
        git config --global --add /home/cchq/www/monolith/releases/2022-09-14_08.59...00549f6cebdb3a86d9222bd8d5db42495d76b4dd'"'"'"'"[75/1951]
 --minutes 13 --commit 00549f6cebdb3a86d9222bd8d5db42495d76b4dd'"'"''
Ubuntu 18.04.6 LTS
Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(255)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/", line 208, in <module>
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/", line 45, in main
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/", line 419, in execute_fro
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/", line 413, in execute
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/corehq/apps/hqadmin/management/commands/", line 64, in handle
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 776, in save_base
    updated = self._save_table(
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)                                                               [43/1951]
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 919, in _do_insert
    return manager._insert(
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/sql/", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/", line 75, in _execute_with_wrapp
    return executor(sql, params, many, context)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(255)

Connection to closed.

Preparing to deploy Formplayer to: monolith

>> Compiling deploy summary
Error getting current formplayer version: HTTPConnectionPool(host='', port=8081): Max retries exceeded with url: /info (Caused by NewConnectionError('
<urllib3.connection.HTTPConnection object at 0x7f7b1d4b7f60>: Failed to establish a new connection: [Errno 111] Connection refused',))
New version details:
    Release Name        : 2022-09-14_10.04
    Commit              : b39109821cfd92abc126320dbb6742c423efe974
        Commit message      : updating files: scripts/staging.yaml
    Commit date         : 5 days ago (2022-09-08T11:32:44+0000)
    Build time          : 7 hours ago (2022-09-14T02:17:34.120Z)

I then say yes to continue with the deploy and get this output:

Continue with deploy? [y/N]y                                                                                                                          [2/1951]
>> Sending email: ATTENTION: ccc has initiated a Formplayer deploy to monolith with non-default branch '00549f6cebdb3a86d9222bd8d5db42495d76b4dd'
Ubuntu 18.04.6 LTS
Connection to closed.
Traceback (most recent call last):
  File "/home/ccc/.virtualenvs/cchq/bin/cchq", line 33, in <module>
    sys.exit(load_entry_point('commcare-cloud', 'console_scripts', 'cchq')())
  File "/home/ccc/commcare-cloud/src/commcare_cloud/", line 262, in main
    exit_code = call_commcare_cloud()
  File "/home/ccc/commcare-cloud/src/commcare_cloud/", line 231, in call_commcare_cloud
    exit_code =, unknown_args)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/", line 102, in run
    rc = deploy_formplayer(environment, args)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/", line 71, in deploy_formplayer
    rc = run_ansible_playbook_command(environment, args)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/", line 126, in run_ansible_playbook_command
  File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/", line 383, in create_generated_yml
  File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/", line 20, in _memoized
    cache[key] = value = fn(*args, **kwargs)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/", line 148, in postgresql_config
    postgresql_config = PostgresqlConfig.wrap(postgresql_json)
  File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/schemas/", line 95, in wrap
    PostgresqlOverride.wrap(data.get('postgres_override', {}))
  File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
  File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.__init__
  File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
  File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
  File "jsonobject/base_properties.pyx", line 278, in jsonobject.base_properties.AssertTypeProperty.wrap
  File "jsonobject/base_properties.pyx", line 269, in jsonobject.base_properties.AssertTypeProperty.assert_type
jsonobject.exceptions.BadValueError: 10 not of type (<class 'str'>, <class 'str'>)

...out of interest a django-manage check_services produces positive output:

(cchq) ccc@monolith:~/commcare-cloud$ cchq monolith django-manage check_services
ssh ccc@ -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python check_services'"'"''
Ubuntu 18.04.6 LTS
SUCCESS (Took   0.61s) kafka          : Kafka seems to be in order
SUCCESS (Took   0.00s) redis          : Redis is up and using 35.67M memory
SUCCESS (Took   0.02s) postgres       : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK Successfully got a user from postgres
SUCCESS (Took   0.01s) couch          : Successfully queried an arbitrary couch view
SUCCESS (Took   0.00s) celery         : OK
SUCCESS (Took   0.02s) elasticsearch  : Successfully sent a doc to ES and read it back
SUCCESS (Took   0.03s) blobdb         : Successfully saved a file to the blobdb
SUCCESS (Took   0.33s) formplayer     : Formplayer returned a 200 status code:
SUCCESS (Took   0.00s) rabbitmq       : RabbitMQ OK
Connection to closed.
You have new mail in /var/mail/ccc

Hi Ed

There are two separate issues here. I'll address the first one here and the 2nd one I'll address in Issue with changelog 53. Upgrade Redis

It looks like the issues was with commcare-cloud getting the deploy diff URL. I suspect there is something wrong with the current release. If you run git rev-parse HEAD in the current release I think you'll get the same error.

I think this happens when the files in the git repo have mixed ownership. You could try run:

/home/cchq/monolith/releases $ chown -R cchq *
1 Like

Thanks Simon - the changelog 53 issue resolution solved the second error when deploying so the deploy succeeds.

The earlier dubious ownership error is still there after chown of the /home/cchq/www/monolith/releases directory, files and subdirectories. I'm not sure if that's affecting anything in the deploy though, it does say it was successful after completing.