I'm unable to get the Kafka service up on a server after it ran out of space.
This particular server was configured with the /opt/data directory being on the same drive as the OS.
When it ran out of space, I, as a temporary measure, copied the contents /opt/data (encrypted files) to a new drive mounted elsewhere, renamed /opt/data to /opt/data.old, cleared up some space, and finally created a directory symlink in /opt/data to the new location. I then ran after_reboot all and the system seemed to come up fine except for Kafka. Not sure if the Kafka issue is related, but I'm assuming so.
my Kafka server.log looks like this at the tail:
[2022-06-14 08:06:24,870] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,871] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,871] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,871] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,872] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,872] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,873] INFO Property zookeeper.connect is overridden to xxx.xxx.xxx.xxx:2181 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,873] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2022-06-14 08:06:24,945] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2022-06-14 08:06:24,950] INFO [Kafka Server 0], Connecting to zookeeper on xxx.xxx.xxx.xxx:2181 (kafka.server.KafkaServer)
The Kafka server.out log has a series of these messages being inserted every second or so:
[2022-06-14 08:11:58,870] INFO Opening socket connection to server xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-06-14 08:11:58,870] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
The controller.log file shows no new logs for today
I imagine this is a zookeeper issue - where should I be looking for those logs?
Any help is appreciated!
EDIT trying some of the suggestions here:
Running this:
cd /opt/kafka/bin
./zookeeper-shell.sh localhost:2181
Results in a series of errors:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
[2022-06-14 08:28:28,383] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2022-06-14 08:28:29,485] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2022-06-14 08:28:30,590] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
... etc
Output of lsof -i -P -n | grep LISTEN indicates neither zookeeper or kafka are listening for incoming connections.