Are there sync logs available from the mobile application?
Do you mean logs of the sync attempts? I don't think these are available though I can't think what you'd need them for.
Re kafka not working, have you confirmed that Zookeeper is running? I assume you've checked in the Kafka logs for any useful output (/opt/kafka/logs/controller.log).
It might also be useful to check the broker details in Zookeeper:
$ cd /opt/kafka/bin
$ ./zookeeper-shell.sh localhost:2181
> ls /brokers/ids
[1, 2]
> get /brokers/ids/1
{"endpoints":["PLAINTEXT://10.xxx.xx.170:9092"],"host":"10.xxx.xx.170"}
> get /brokers/ids/2
{"endpoints":["PLAINTEXT://10.xxx.xx.171:9092"],"host":"10.xxx.xx.171"}
Thanks a lot for your reply. We're still struggling go to get all the services up and running. Here's the output of the commands you recommended. Any clue?
Switched localhost to FDQN in the server.properties file, added listener, start/stop of Zookeeper, restarted Kafka and after-reboot of Kafka. Still, no brokers are available.
It looks like using the FDQN worked. The shell doesn't give you a prompt, it's just bare. You could try type "?" And see if you get the help output.
I think this is making sense. If there is a mismatch between what the service is binding to and what the client is trying to connect to the it could cause these types of issues.
You could check your Kafka and zookeeper config to see what the network binds are. Also check how you have things configured in your commcare-cloud inventory. I would recommend using the host's private IP addresses rather than the FDQN.