Apache Kafka

Set Up Logger

Add a logger to receive Apache Kafka messages as shown below:

• Kafka Broker Address List: Comma-separated list in host:port format
• Kafka topic: Name of the Kafka topic to subscribe to
• Kafka Group ID: Kafka consumer group ID (use the same ID across loggers to balance topic load)
• Security Protocol: Select the protocol to use for log transmission. The default is PLAINTEXT.
  • PLAINTEXT: No additional configuration is required.
  • SSL (One-way): Enter the SSL Truststore path, password, and type. In addition, add the ssl.client.auth=none option to the Kafka server configuration file (config/server.properties).
  • SSL (Mutual Authentication): Enter the SSL Truststore and Keystore paths, passwords, and types, as well as the SSL Key password and SSL Endpoint Identification Algorithm.

Logstash Integration

Use the following examples to push data into Kafka topics from Logstash.

Plain Codec Example

Send plain text messages to the test topic:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { kafka { codec => plain { format => "%{message}" } topic_id => "test" } }'

JSON Codec Example

Send JSON-formatted messages to the test topic:

/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { kafka { codec => json { format => "%{message}" } topic_id => "test" } }'

To process structured messages, stream data with the json codec and use the parsejson to extract fields easily.