So, I've fenced a problem after monolith install, followed by this guide:
https://commcare-cloud.readthedocs.io/en/latest/installation/2-manual-install.html
After successful install the next step gets error: commcare-cloud monolith django-manage create_kafka_topics
Ubuntu 18.04.6 LTS
Traceback (most recent call last):
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/manage.py", line 190, in
main()
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/manage.py", line 44, in main
execute_from_command_line(sys.argv)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/corehq/apps/change_feed/management/commands/create_kafka_topics.py", line 10, in handle
create_kafka_topics()
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/corehq/apps/change_feed/management/commands/create_kafka_topics.py", line 14, in create_kafka_topics
with get_simple_kafka_client() as client:
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/corehq/apps/change_feed/connection.py", line 12, in get_simple_kafka_client
return ClosingContextProxy(SimpleClient(
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/kafka/client.py", line 57, in init
self.load_metadata_for_topics() # bootstrap with all metadata
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/kafka/client.py", line 525, in load_metadata_for_topics
resp = self.send_metadata_request(topics)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/kafka/client.py", line 590, in send_metadata_request
return self._send_broker_unaware_request(payloads, encoder, decoder)
File "/home/cchq/www/monolith/releases/2022-06-24_20.34/python_env/lib/python3.9/site-packages/kafka/client.py", line 177, in _send_broker_unaware_request
raise KafkaUnavailableError('All servers failed to process request: %s' % (hosts,))
kafka.errors.KafkaUnavailableError: KafkaUnavailableError: All servers failed to process request: [('192.169.233.136', 9092, <AddressFamily.AF_INET: 2>)]
Connection to 192.169.233.136 closed.
The commands commcare-cloud monolith django-manage check_services results:
Ubuntu 18.04.6 LTS
FAILURE (Took 0.05s) kafka : Could not connect to Kafka: NoBrokersAvailable
SUCCESS (Took 0.02s) redis : Redis is up and using 809.15K memory
FAILURE (Took 0.11s) postgres : default:commcarehq:OK p1:commcarehq_p1:OK p2:commcarehq_p2:OK proxy:commcarehq_proxy:OK synclogs:commcarehq_synclogs:OK ucr:commcarehq_ucr:OK No users found in postgres
SUCCESS (Took 0.14s) couch : Successfully queried an arbitrary couch view
FAILURE (Took 0.00s) celery : analytics_queue has been blocked as long as we can see (max allowed is 0:30:00)
async_restore_queue has been blocked as long as we can see (max allowed is 0:01:00)
background_queue has been blocked as long as we can see (max allowed is 0:10:00)
case_import_queue has been blocked as long as we can see (max allowed is 0:01:00)
case_rule_queue has been blocked as long as we can see (max allowed is 1:00:00)
celery has been blocked as long as we can see (max allowed is 0:01:00)
celery_periodic has been blocked as long as we can see (max allowed is 0:10:00)
email_queue has been blocked as long as we can see (max allowed is 0:00:30)
export_download_queue has been blocked as long as we can see (max allowed is 0:00:30)
repeat_record_queue has been blocked as long as we can see (max allowed is 1:00:00)
saved_exports_queue has been blocked as long as we can see (max allowed is 6:00:00)
ucr_queue has been blocked as long as we can see (max allowed is 1:00:00)
EXCEPTION (Took 0.10s) elasticsearch : Service check errored with exception 'ESError(NotFoundError(404, 'index_not_found_exception', {'error': {'root_cause': [{'type': 'index_not_found_exception', 'reason': 'no such index', 'resource.type': 'index_or_alias', 'resource.id': 'hqgroups_2017-05-29', 'index': 'hqgroups_2017-05-29'}], 'type': 'index_not_found_exception', 'reason': 'no such index', 'resource.type': 'index_or_alias', 'resource.id': 'hqgroups_2017-05-29', 'index': 'hqgroups_2017-05-29'}, 'status': 404}))'
SUCCESS (Took 0.25s) blobdb : Successfully saved a file to the blobdb
FAILURE (Took 0.02s) formplayer : Formplayer returned a 502 status code: https://capi.gstat.gov.ua/formplayer/serverup
SUCCESS (Took 0.00s) rabbitmq : RabbitMQ OK
Connection to 192.169.233.136 closed.
After analyzing commcare-cloud monolith service kafka status:
192.169.233.136 | CHANGED | rc=0 >>
● kafka-server.service - Apache Kafka server (broker)
Loaded: loaded (/etc/systemd/system/kafka-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-06-25 13:44:56 UTC; 11min ago
Docs: Apache Kafka
Main PID: 11686 (java)
Tasks: 59 (limit: 19660)
CGroup: /system.slice/kafka-server.service
└─11686 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -Dkafka.logs.dir=/opt/kafka/bin/../logs -Dlog4j.configuration=file:/opt/kafka/bin/../config/log4j.properties -cp /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.6.1.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.6.1.jar:/opt/kafka/bin/../libs/connect-file-2.6.1.jar:/opt/kafka/bin/../libs/connect-json-2.6.1.jar:/opt/kafka/bin/../libs/connect-mirror-2.6.1.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.6.1.jar:/opt/kafka/bin/../libs/connect-runtime-2.6.1.jar:/opt/kafka/bin/../libs/connect-transforms-2.6.1.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.10.2.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.25.0-GA.jar:/opt/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.31.jar:/opt/kafka/bin/../libs/jersey-common-2.31.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.31.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.31.jar:/opt/kafka/bin/../libs/jersey-hk2-2.31.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.31.jar:/opt/kafka/bin/../libs/jersey-server-2.31.jar:/opt/kafka/bin/../libs/jetty-client-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-http-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-io-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-security-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-server-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jetty-util-9.4.33.v20201020.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka_2.13-2.6.1.jar:/opt/kafka/bin/../libs/kafka_2.13-2.6.1-sources.jar:/opt/kafka/bin/../libs/kafka-clients-2.6.1.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.6.1.jar:/opt/kafka/bin/../libs/kafka-streams-2.6.1.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.6.1.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-2.6.1.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.6.1.jar:/opt/kafka/bin/../libs/kafka-tools-2.6.1.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.50.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.50.Final.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.1.6.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/opt/kafka/bin/../libs/scala-library-2.13.2.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.2.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/zookeeper-3.5.8.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.8.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.4-7.jar kafka.Kafka /etc/kafka/config/server.properties
Jun 25 13:56:08 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:08,677] INFO Opening socket connection to server capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:08 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:08,678] INFO Socket error occurred: capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:09 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:09,778] INFO Opening socket connection to server capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:09 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:09,779] INFO Socket error occurred: capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:10 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:10,880] INFO Opening socket connection to server capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:10 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:10,880] INFO Socket error occurred: capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:11 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:11,981] INFO Opening socket connection to server capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:11 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:11,982] INFO Socket error occurred: capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:13 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:13,084] INFO Opening socket connection to server capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Jun 25 13:56:13 capi.gstat.gov.ua kafka-server-start.sh[11686]: [2022-06-25 13:56:13,084] INFO Socket error occurred: capi.gstat.gov.ua.capi.gstat.gov.ua/192.169.233.136:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
ansible zookeeper -m shell -i /home/lamp/environments/monolith/inventory.ini -a 'service zookeeper status' --diff -u ansible --become -e @/home/lamp/environments/monolith/public.yml -e @/home/lamp/environments/monolith/.generated.yml -e @/home/lamp/environments/monolith/vault.yml --vault-password-file=/home/lamp/commcare-cloud/src/commcare_cloud/ansible/echo_vault_password.sh '--ssh-common-args=-o UserKnownHostsFile=/home/lamp/environments/monolith/known_hosts'
[WARNING]: Consider using the service module rather than running 'service'. If you need to use command because service
is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid
of this message.
192.169.233.136 | FAILED | rc=4 >>
Unit zookeeper.service could not be found.non-zero return cod
It seems like there are problem with zookeeper.service
When reboot system and execute commcare-cloud monolith after-reboot all there are some warnings like this at start
[WARNING]: Could not match supplied host pattern, ignoring: pgbouncer
PLAY [CitusDB] **********************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: citusdb_master
PLAY [pgbouncer] ********************************************************************************************************
skipping: no hosts matched
PLAY [Setup auth for standby] *******************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: citusdb_worker
TASK [start monit] ******************************************************************************************************
ok: [192.169.233.136]
[WARNING]: flush_handlers task does not support when conditional
TASK [start the redis service] ******************************************************************************************
changed: [192.169.233.136]
[WARNING]: Could not match supplied host pattern, ignoring: redis_master
PLAY [configure the master redis server] ********************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: redis_slave
PLAY [configure redis slaves] *******************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: redis_sentinel
PLAY [configure redis sentinel nodes] ***********************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: keepalived_redis
Though, as I'm new to commcare, any suggestions would be appreciated.