Redeployment error: echis

Developers
1

i repeatedly try to redeploy echis but failed on this step. i am not able to figure out the problem

[172.19.3.41] sudo: sudo -S -p 'sudo password:' -H -u "root" nginx -s reload
Traceback (most recent call last):
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/main.py", line 749, in main
*args, **kwargs
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 550, in deploy_commcare
_deploy_without_asking(skip_record)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 406, in _deploy_without_asking
deploy_checkpoint(index, command.name, execute_with_timing, command)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 399, in deploy_checkpoint
fn(*args, **kwargs)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/fabfile.py", line 389, in conditionally_stop_pillows_and_celery_during_migrate
execute_with_timing(db.migrate)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/utils.py", line 25, in execute_with_timing
execute(fn, *args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 385, in execute
multiprocessing
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 275, in _execute
return task.run(*args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/tasks.py", line 172, in run
return self.wrapped(*args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/decorators.py", line 52, in inner_decorator
return func(*args, **kwargs)
File "/home/ansible/commcare-cloud/src/commcare_cloud/fab/operations/db.py", line 96, in migrate
sudo('%(py3_virtualenv_root)s/bin/python manage.py migrate_multi --noinput' % env)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/network.py", line 654, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 1121, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/operations.py", line 952, in _run_command
error(message=msg, stdout=out, stderr=err)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 359, in error
return func(message)
File "/home/ansible/.virtualenvs/cchq/lib/python3.6/site-packages/fabric/utils.py", line 55, in abort
raise env.abort_exception(msg)
Exception: sudo() received nonzero return code 1 while executing!
Requested: /home/cchq/www/echis/releases/2021-06-17_04.58/python_env-3.6/bin/python manage.py migrate_multi --noinput
Executed: sudo -S -p 'sudo password:' -H -u "cchq" /bin/bash -l -c "cd /home/cchq/www/echis/releases/2021-06-17_04.58 >/dev/null && /home/cchq/www/echis/re
leases/2021-06-17_04.58/python_env-3.6/bin/python manage.py migrate_multi --noinput"

2

you could try running that step manually from the release to see if it gives more details:

  1. SSH to the 'django-manage' machine:
cchq echis ssh django-manage
  1. Run the command
sudo -iu cchq
/home/cchq/www/echis/re
leases/2021-06-17_04.58/python_env-3.6/bin/python manage.py migrate_multi --noinput
3

Here is the output:

detailed log output file: populate_user_role_2021-06-17T07-56-59.log
Found 48 UserRole docs and 0 SQLUserRole models
Processing Started at 2021-06-17 07:56:59
Processing [ ] 0/48 0% -:--:-- remaining
Processing [. ] 1/48 2% 0:00:08 remaining
Processing [.... ] 4/48 8% 0:00:03 remaining
Processing [..... ] 5/48 10% 0:00:03 remaining
Processing [........ ] 8/48 17% 0:00:03 remaining
Processing [............ ] 12/48 25% 0:00:02 remaining
Processing [............. ] 13/48 27% 0:00:02 remaining
Processing [................ ] 16/48 33% 0:00:02 remaining
Processing [................... ] 19/48 40% 0:00:01 remaining
Processing [....................... ] 23/48 48% 0:00:01 remaining
Processing [........................... ] 27/48 56% 0:00:01 remaining
Processing [............................... ] 31/48 65% 0:00:01 remaining
Processing [................................... ] 35/48 73% 0:00:01 remaining
Processing [....................................... ] 39/48 81% 0:00:00 remaining
Processing [........................................... ] 43/48 90% 0:00:00 remaining
Processing [.............................................. ] 46/48 96% 0:00:00 remaining
Processing [.............................................. ] 46/48 96% 0:00:01.787716 elapsed
Processing Finished at 2021-06-17 07:57:01
Processing Elapsed time: 0:00:02
Processed 46 documents
Found 0 differences
Automatic migration failed, 4 items remain to migrate.

            A migration must be performed before this environment can be upgraded to the latest version
            of CommCareHQ. This migration is run using the management command populate_user_role.


            Run the following commands to run the migration and get up to date:

                commcare-cloud <env> fab setup_limited_release --set code_branch=TODO: add once the PR adding this file is merged

                commcare-cloud <env> django-manage --release <release created by previous command> populate_user_role

                commcare-cloud <env> deploy commcare
4

Hi Demisew

I'm not sure why there are 4 roles that remain unmigrated. You can re-run the migration as follows:

(run from the same release on the 'django-manage' machine as the cchq user)

/home/cchq/www/echis/re
leases/2021-06-17_04.58/python_env-3.6/bin/python manage.py populate_user_role

If there are still unmigrated roles we'll have to do some debugging to figure out why.

5

(python_env-3.6) (echis) cchq@echis0:~/www/echis/releases/2021-06-17_04.58$ ./manage.py populate_user_role

Detailed log output file: populate_user_role_2021-06-17T09-36-10.log
Found 48 UserRole docs and 0 SQLUserRole models
Processing Started at 2021-06-17 09:36:10
Processing [ ] 0/48 0% -:--:-- remaining
Processing [. ] 1/48 2% 0:00:30 remaining
Processing [.. ] 2/48 4% 0:00:00.632673 elapsed
Traceback (most recent call last):
File "./manage.py", line 195, in
main()
File "./manage.py", line 42, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/home/cchq/www/echis/releases/2021-06-17_04.58/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/corehq/apps/cleanup/management/commands/populate_sql_model_from_couch_model.py", line 246, in handle
self._migrate_doc(doc, logfile)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/corehq/apps/cleanup/management/commands/populate_sql_model_from_couch_model.py", line 271, in _migrate_doc
model, created = self.update_or_create_sql_object(doc)
File "/home/cchq/www/echis/releases/2021-06-17_04.58/corehq/apps/users/management/commands/populate_user_role.py", line 63, in update_or_create_sql_object
couch_id=doc['_id'],
TypeError: 'NoneType' object is not subscriptable
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

6

Hi Simon,

Now, i am getting different output after updating couch_view

Found 53 UserRole docs and 1 SQLUserRole models
Processing Started at 2021-06-17 09:48:27
Processing [ ] 0/53 0% -:--:-- remaining
Processing [. ] 1/53 2% 0:00:08 remaining
Processing [.. ] 2/53 4% 0:00:07 remaining
Processing [.... ] 4/53 8% 0:00:05 remaining
Processing [...... ] 6/53 11% 0:00:05 remaining
Processing [........ ] 8/53 15% 0:00:05 remaining
Processing [......... ] 10/53 19% 0:00:04 remaining
Processing [........... ] 12/53 23% 0:00:04 remaining
Processing [.............. ] 15/53 28% 0:00:03 remaining
Processing [............... ] 16/53 30% 0:00:03 remaining
Processing [................. ] 18/53 34% 0:00:03 remaining
Processing [................... ] 20/53 38% 0:00:03 remaining
Processing [..................... ] 22/53 42% 0:00:03 remaining
Processing [....................... ] 24/53 45% 0:00:02 remaining
Processing [......................... ] 26/53 49% 0:00:02 remaining
Processing [........................... ] 29/53 55% 0:00:02 remaining
Processing [............................. ] 31/53 58% 0:00:02 remaining
Processing [............................... ] 33/53 62% 0:00:01 remaining
Processing [................................. ] 35/53 66% 0:00:01 remaining
Processing [................................... ] 37/53 70% 0:00:01 remaining
Processing [..................................... ] 39/53 74% 0:00:01 remaining
Processing [....................................... ] 41/53 77% 0:00:01 remaining
Processing [......................................... ] 43/53 81% 0:00:01 remaining
Processing [.......................................... ] 45/53 85% 0:00:01 remaining
Processing [............................................ ] 47/53 89% 0:00:00 remaining
Processing [.............................................. ] 49/53 92% 0:00:00 remaining
Processing [................................................ ] 51/53 96% 0:00:00 remaining
Processing [..................................................] 53/53 100% 0:00:03.870777 elapsed
Processing [..................................................] 53/53 100% 0:00:03.887109 elapsed
Processing Finished at 2021-06-17 09:48:31
Processing Elapsed time: 0:00:04
Processed 53 documents
Found 0 differences

7

Hi Demisew

It's very odd that you'd get different results from the couch view. What do you mean by after updating couch_view?

8

on couchdb goto commcarehq__users db --> view --->build and save document

But the build is still failed.