Deploy-stack script issues

#1

During a fresh monolith deployment on Ubuntu 18.04.3 LTS, when running the deploy-stack scripts, the system consistently halts on:

TASK [pgbouncer : pgbouncer users] **********************************************************************************************************************************************************
fatal: [192.168.253.21]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: Unable to look up a name or access an attribute in template string ({% for group in postgres_users.values() %}\n\"{{ group.username }}\" \"{{ group.password }}\"\n{% endfor %}\n{% if inventory_hostname in groups['citusdb_worker'] %}\n\"postgres\" \"\"\n{% endif %}\n).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'AnsibleUndefined' is not iterable"}

Where are these group.username and group.password variables meant to be? Also, I don’t see reference to citusdb_worker in any of the config files.

My inventory.ini is as folows:

[monolith]
192.168.253.21

[monolith:vars]
elasticsearch_node_name=es0
kafka_broker_id=0
hostname='monolith'
ufw_private_interface=eth0

[control:children]
monolith

[proxy:children]
monolith

[webworkers:children]
monolith

[celery:children]
monolith

[pillowtop:children]
monolith

[formplayer:children]
monolith

[django_manage:children]
monolith

[postgresql:children]
monolith

[pg_backup:children]
monolith

[pg_standby]

[couchdb2:children]
monolith

[couchdb2_proxy:children]
monolith

[redis:children]
monolith

[zookeeper:children]
monolith

[kafka:children]
monolith

[elasticsearch:children]
monolith

[rabbitmq:children]

[shared_dir_host:children]

[riakcs]

Any assistance would be appreciated!

#3

Hi Ed. This should fix it: https://github.com/dimagi/commcare-cloud/pull/3441

1 Like
#4

Thanks for the prompt response. I’ll get latest. Adding [citusdb_worker] to the end of inventory.ini with an empty array resolved it on my end but I’ll get latest regardless and test.
Thanks!

#5

Hi, it’s now failing with:

TASK [Configure Pl/Proxy cluster] ***********************************************************************************************************************************************************
fatal: [192.168.253.21]: FAILED! => {“changed”: false, “cmd”: “./manage.py configure_pl_proxy_cluster --create_only”, “msg”: “\n:stderr: /home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use “pip install psycopg2-binary” instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.\n “””)\n2019-12-05 14:47:27,600 INFO AXES: BEGIN LOG\nTraceback (most recent call last):\n File “./manage.py”, line 195, in \n main()\n File “./manage.py”, line 41, in main\n execute_from_command_line(sys.argv)\n File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line\n utility.execute()\n File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/init.py”, line 356, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py”, line 283, in run_from_argv\n self.execute(*args, **cmd_options)\n File “/home/cchq/www/monolith/current/python_env-3.6/lib/python3.6/site-packages/django/core/management/base.py”, line 330, in execute\n output = self.handle(*args, **options)\n File “/home/cchq/www/monolith/releases/2019-12-05_13.17/corehq/sql_db/management/commands/configure_pl_proxy_cluster.py”, line 55, in handle\n create_or_update_cluster(plproxy_standby_config, verbose, options[‘create_only’])\n File “/home/cchq/www/monolith/releases/2019-12-05_13.17/corehq/sql_db/management/commands/configure_pl_proxy_cluster.py”, line 59, in create_or_update_cluster\n existing_config = _get_existing_cluster_config(cluster_config)\n File “/home/cchq/www/monolith/releases/2019-12-05_13.17/corehq/sql_db/management/commands/configure_pl_proxy_cluster.py”, line 162, in _get_existing_cluster_config\n proxy_db = cluster_config.proxy_db\nAttributeError: ‘NoneType’ object has no attribute ‘proxy_db’\n", “path”: “/home/cchq/www/monolith/current/python_env-3.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin”, “syspath”: ["/tmp/ansible_django_manage_payload_uYrb2T/ansible_django_manage_payload.zip", “”, “/usr/lib/python2.7”, “/usr/lib/python2.7/plat-x86_64-linux-gnu”, “/usr/lib/python2.7/lib-tk”, “/usr/lib/python2.7/lib-old”, “/usr/lib/python2.7/lib-dynload”, “/usr/local/lib/python2.7/dist-packages”, “/usr/lib/python2.7/dist-packages”]}

What stands out to me is the “AttributeError: ‘NoneType’ object has no attribute ‘proxy_db’”

Any assistance appreciated

#6

Hi Ed, thanks for finding the bug. This was recently introduced by some changes I’ve been making for large scale projects.

I’ve PR’d a fix here: https://github.com/dimagi/commcare-hq/pull/26126

If you’re setting up a monolith you could just update the code in /home/cchq/www/<env>/current so that you aren’t blocked until the PR is merged and then re-run the Ansible playbook.

Alternatively if you wait for the change to be merged it should get picked up when you re-run the playbook (assuming you’re running it with -e 'CCHQ_IS_FRESH_INSTALL=1'.

#7

Awesome, thanks for the quick response Simon!
Ed