Kafka Error: Wrong request type

Developers
1

we are using 2 nodes Kafka servers on the cluster to handle the load. both of them are running. I recently noticed there is an error on the log.

Environment:
* ubuntu 14.04 and ubuntu 18.04,
* both running the same version of Kafka-server

tail -f /opt/kafka/logs/server.log

[2020-01-27 21:40:09,292] INFO Closing socket connection to /172.19.4.42. (kafka.network.Processor)
[2020-01-27 21:40:09,293] ERROR Closing socket for /172.19.4.42 because of error (kafka.network.Processor)
kafka.common.KafkaException: Wrong request type 18
at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
at kafka.network.RequestChannel$Request.(RequestChannel.scala:50)
at kafka.network.Processor.read(SocketServer.scala:450)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:748)
[2020-01-27 21:40:09,395] ERROR Closing socket for /172.19.4.42 because of error (kafka.network.Processor)
kafka.common.KafkaException: Wrong request type 16
at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:64)
at kafka.network.RequestChannel$Request.(RequestChannel.scala:50)
at kafka.network.Processor.read(SocketServer.scala:450)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:748)

2

What version of Kafka are you running?

There's some info on this error online: apache kafka - KafkaException: Wrong request type 18 - Stack Overflow

3

Version: 2.10-0.8.2.2, on both nodes.
But the OS is: Ubuntu 18.04 and ubuntu 14.04

I've see the link you shared before, and still am not able to figure the cause.

Here we are using replication-factor of 2 and each topic has either 5 or 10 partitions.
The fowllowing is an example for form-sql topic.

kafka-topics.sh --zookeeper 172.19.3.40:2181 --describe --topic form-sql
output:

Topic:form-sql PartitionCount:10 ReplicationFactor:2 Configs:
Topic: form-sql Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 3 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 4 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 6 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 7 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 8 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: form-sql Partition: 9 Leader: 0 Replicas: 0,1 Isr: 0,1

The following is an example of app topic. we created five partition for it.

kafka-topics.sh --zookeeper 172.19.3.40:2181 --describe --topic app

Output
Topic:app PartitionCount:6 ReplicationFactor:2 Configs:
Topic: app Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: app Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: app Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: app Partition: 3 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: app Partition: 4 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: app Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1

4

Hi demis08,

According to that StackOverflow page, the error is happening because the broker is getting requests that it can't understand.

So if the broker and the client are definitely running the same version, then that can't be the cause. Something other than a client must be sending those requests to the broker.

It could be monitoring software. Or maybe an anti-malware scanner. Or a penetration tester.

I don't know exactly how your environment is configured, so I'm not sure whether any of those are possibilities. Could it be monitoring software?

There is something about your logs that makes me think it's not monitoring software or an anti-malware scanner:

kafka.common.KafkaException: Wrong request type 18
...
kafka.common.KafkaException: Wrong request type 16

The requests have specific request types. So I suspect they are coming from a client (or something pretending to be a client but most probably a client).

I'm not sure that gives you much more to go on, but I hope it helps.

1 Like
5

I think I tracked these down. These are coming from the Kafka client which is sending requests to the borker to try and list groups and get version info:

I think these are harmless

1 Like