Apache Kafka

Set Up Logger

Add a logger to receive Apache Kafka messages as shown below:

• Kafka Broker Address List: Comma-separated list in host:port format
• Kafka topic: Name of the Kafka topic to subscribe to
• Kafka Group ID: Kafka consumer group ID (use the same ID across loggers to balance topic load)

Logstash Integration

Use the following examples to push data into Kafka topics from Logstash.

Plain Codec Example

Send plain text messages to the test topic:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { kafka { codec => plain { format => "%{message}" } topic_id => "test" } }'

JSON Codec Example

Send JSON-formatted messages to the test topic:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { kafka { codec => json { format => "%{message}" } topic_id => "test" } }'

To process structured messages, stream data with the json codec and use the parsejson to extract fields easily.