sionsmith/kafka-consumer-seek-example repository overview

⁠Sample Kafka producer and consumer using Spring Framework

⁠How to run the sample applications

1. Start up the kafka stack by running in the docker directory.

   $ docker-compose -f ./docker/single-broker.yml up 
   

   Copy
2. Install the Producer with Maven.

   $ mvn -f java/producer/pom.xml clean install 
   

   Copy
3. Run the Producer. This loads some sample data from an sample_event.json

   $ java -jar java/producer/target/demo-producer-0.0.1-SNAPSHOT.jar
   

   Copy
4. Install the Consumer with Maven.

   $ mvn -f java/consumer/pom.xml clean install 
   

   Copy
5. Run the Consumer. The consumer reads a single event from a given offset and partition. This payload is then sent to a lambda function.

   $ export AWS_ACCESS_KEY_ID=xxx
   $ export AWS_SECRET_ACCESS_KEY=xxx
   $ export SPRING.KAFKA.CONSUMER.BOOTSTRAP-SERVERS=127.0.0.1:9092
   $ export SPRING.KAFKA.CONSUMER.ERROR-TOPIC-NAME=shipment-sink-error
   $ export LAMBDA.REGION=eu-west-2
   $ export LAMBDA.FUNCTION-NAME=shipment-process-dev-index
   # run the consumer with the following
   $ java -jar java/consumer/target/demo-consumer-0.0.1-SNAPSHOT.jar
   

   Copy
6. Destroy the environment

   $ docker-compose -f ./docker/single-broker.yml down -v
   

   Copy

⁠Kafka configuration

The producer and consumer configs can be set inside application.yaml or overriden via environmental variables as shown above.

Reference - how Spring Boot processes external configuration⁠

Reference for all available configuration parameters of the Kafka consumer and producer:

• Producer configs⁠
• Consumer configs⁠

⁠Appendix

⁠Useful commands

• List out topics in the cluster kafka-topics --bootstrap-server localhost:9092 --list

• Send a test event to the error sink topic jq -rc . ./error_event.json | kafka-console-producer --broker-list localhost:9092 --topic shipment-sink-error