Some time you might have a bad record in Kafka topic that you want to delete. Kafka does not provide direct option to delete specific record. Only way to delete records is to expire them. You can achieve this by setting data retention to say 1 second that expires all the old messages. You can follow these steps
retention.ms
config parameter for the topic
kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topicsConfigs for topic 'my-topic' are retention.ms=86400000
retention.ms
to 1 which means all messages older than 1 ms will be expired
kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=1Completed Updating config for entity: topic 'my-topic'
retention.ms
back to its original value which was 86400000 (7 days)
kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=86400000Completed Updating config for entity: topic 'my-topic'.
kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topicsConfigs for topic 'my-topic' are retention.ms=86400000