Hi all, it seems the migration went well but I've since been unable to deploy. It seems no matter which release I deploy, I get the following error consistently - starting at the end of the test phase:
[10.0.0.4] Executing task 'record_successful_release'
[10.0.0.4] sudo: echo '/home/cchq/www/monolith/releases/2022-09-14_09.50' >> "$(echo RELEASES.txt)"
Done.
Disconnecting from 10.0.0.4... done.
>> Sending email: CommCare HQ deploy successful - monolith
Ubuntu 18.04.6 LTS
Connection to 10.0.0.4 closed.
commcare-cloud monolith django-manage record_deploy_success --user ccc --environment monolith --url 'https://github.com/dimagi/commcare-hq/compare/fatal: det$
cted dubious ownership in repository at '"'"'/home/cchq/www/monolith/releases/2022-09-14_08.59'"'"'
To add an exception for this directory, call:
git config --global --add safe.directory /home/cchq/www/monolith/releases/2022-09-14_08.59...00549f6cebdb3a86d9222bd8d5db42495d76b4dd' --minutes 13 -$
commit 00549f6cebdb3a86d9222bd8d5db42495d76b4dd
ssh ccc@10.0.0.4 -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_$
nv/bin/python manage.py record_deploy_success --user ccc --environment monolith --url '"'"'"'"'"'"'"'"'https://github.com/dimagi/commcare-hq/compare/fatal: de
tected dubious ownership in repository at '"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'/home/cchq/www/monolith/releases/2022-09-14_08.59'"'"'"'"'"'"'"
'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'
To add an exception for this directory, call:
git config --global --add safe.directory /home/cchq/www/monolith/releases/2022-09-14_08.59...00549f6cebdb3a86d9222bd8d5db42495d76b4dd'"'"'"'"[75/1951]
--minutes 13 --commit 00549f6cebdb3a86d9222bd8d5db42495d76b4dd'"'"''
Ubuntu 18.04.6 LTS
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(255)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/manage.py", line 208, in <module>
main()
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/manage.py", line 45, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_fro
m_command_line
utility.execute()
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/corehq/apps/hqadmin/management/commands/record_deploy_success.py", line 64, in handle
deploy.save()
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
self.save_base(using=using, force_insert=force_insert,
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
updated = self._save_table(
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw) [43/1951]
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
return manager._insert(
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrapp
ers
return executor(sql, params, many, context)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/cchq/www/monolith/releases/2022-09-14_09.50/python_env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(255)
Connection to 10.0.0.4 closed.
Preparing to deploy Formplayer to: monolith
>> Compiling deploy summary
Error getting current formplayer version: HTTPConnectionPool(host='10.0.0.4', port=8081): Max retries exceeded with url: /info (Caused by NewConnectionError('
<urllib3.connection.HTTPConnection object at 0x7f7b1d4b7f60>: Failed to establish a new connection: [Errno 111] Connection refused',))
New version details:
Release Name : 2022-09-14_10.04
Commit : b39109821cfd92abc126320dbb6742c423efe974
Commit message : updating files: scripts/staging.yaml
Commit date : 5 days ago (2022-09-08T11:32:44+0000)
Build time : 7 hours ago (2022-09-14T02:17:34.120Z)
I then say yes to continue with the deploy and get this output:
Continue with deploy? [y/N]y [2/1951]
>> Sending email: ATTENTION: ccc has initiated a Formplayer deploy to monolith with non-default branch '00549f6cebdb3a86d9222bd8d5db42495d76b4dd'
Ubuntu 18.04.6 LTS
Connection to 10.0.0.4 closed.
Traceback (most recent call last):
File "/home/ccc/.virtualenvs/cchq/bin/cchq", line 33, in <module>
sys.exit(load_entry_point('commcare-cloud', 'console_scripts', 'cchq')())
File "/home/ccc/commcare-cloud/src/commcare_cloud/commcare_cloud.py", line 262, in main
exit_code = call_commcare_cloud()
File "/home/ccc/commcare-cloud/src/commcare_cloud/commcare_cloud.py", line 231, in call_commcare_cloud
exit_code = command.run(args, unknown_args)
File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/command.py", line 102, in run
rc = deploy_formplayer(environment, args)
File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/formplayer.py", line 71, in deploy_formplayer
rc = run_ansible_playbook_command(environment, args)
File "/home/ccc/commcare-cloud/src/commcare_cloud/commands/deploy/formplayer.py", line 126, in run_ansible_playbook_command
environment.create_generated_yml()
File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/main.py", line 383, in create_generated_yml
generated_variables.update(self.postgresql_config.to_generated_variables(self))
File "/home/ccc/.virtualenvs/cchq/lib/python3.6/site-packages/memoized.py", line 20, in _memoized
cache[key] = value = fn(*args, **kwargs)
File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/main.py", line 148, in postgresql_config
postgresql_config = PostgresqlConfig.wrap(postgresql_json)
File "/home/ccc/commcare-cloud/src/commcare_cloud/environment/schemas/postgresql.py", line 95, in wrap
PostgresqlOverride.wrap(data.get('postgres_override', {}))
File "jsonobject/base.pyx", line 251, in jsonobject.base.JsonObjectBase.wrap
File "jsonobject/base.pyx", line 202, in jsonobject.base.JsonObjectBase.__init__
File "jsonobject/base.pyx", line 235, in jsonobject.base.JsonObjectBase.set_raw_value
File "jsonobject/base.pyx", line 274, in jsonobject.base.JsonObjectBase.__wrap
File "jsonobject/base_properties.pyx", line 278, in jsonobject.base_properties.AssertTypeProperty.wrap
File "jsonobject/base_properties.pyx", line 269, in jsonobject.base_properties.AssertTypeProperty.assert_type
jsonobject.exceptions.BadValueError: 10 not of type (<class 'str'>, <class 'str'>)
...out of interest a django-manage check_services produces positive output:
(cchq) ccc@monolith:~/commcare-cloud$ cchq monolith django-manage check_services
ssh ccc@10.0.0.4 -t -o UserKnownHostsFile=/home/ccc/environments/monolith/known_hosts 'sudo -iu cchq bash -c '"'"'cd /home/cchq/www/monolith/current; python_env/bin/python manage.py check_services'"'"''
Ubuntu 18.04.6 LTS
SUCCESS (Took 0.61s) kafka : Kafka seems to be in order
SUCCESS (Took 0.00s) redis : Redis is up and using 35.67M memory
SUCCESS (Took 0.02s) postgres : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK Successfully got a user from postgres
SUCCESS (Took 0.01s) couch : Successfully queried an arbitrary couch view
SUCCESS (Took 0.00s) celery : OK
SUCCESS (Took 0.02s) elasticsearch : Successfully sent a doc to ES and read it back
SUCCESS (Took 0.03s) blobdb : Successfully saved a file to the blobdb
SUCCESS (Took 0.33s) formplayer : Formplayer returned a 200 status code: https://xxx.xxx.xxx/formplayer/serverup
SUCCESS (Took 0.00s) rabbitmq : RabbitMQ OK
Connection to 10.0.0.4 closed.
You have new mail in /var/mail/ccc