Best Approach to Consume Kafka Messages in Conductor OSS

Conductor OSS
1

Hi everyone,

Is there a way to consume Kafka messages using Conductor OSS?
I achieve this by creating a sample task using the consumer.poll method to retrieve messages. Is there a better approach for a worker to consume messages efficiently?

Looking forward to your insights!

Thanks!

1 Like
2

Hey @raseldc,

With the OSS setup, there are a couple of ways to consume from Kafka.

However, in my opinion, the best approach is to create a new class or module in your Conductor instance that implements ObservableQueue, similar to how the AMQP module works. You can find an example here.

This will allow you to set up an event handler in Conductor as usual and specify the newly created queue type.

You can follow the AMQP module as a reference but adapt it specifically for Kafka.

Let me know if you need any further assistance!

2 Likes