Kafka
Apache Kafka is a distributed and easily scalable event storage and streaming platform for real time data.
Configuration
The configuration of Kafka datasources is described in detail in the Kafka Datasource Object Guide.
Subscribing to a Kafka Topic
To subscribe to a Kafka topic, IO-Item objects are used.
1 To create a new IO-Item, right-click on the Kafka Datasource object in the I/O Model and select to open the Create Object wizard.
2 In the Available Objects list, select External Interfaces > IO-Item Kafka.
3 In the Property Compounds list, select Common compound and enter an Object Name for the new IO-Item.
4 For the Item ID property, provide the name of the Kafka to which this IO-Item should subscribe. If the topic name begins with '^'' then it becomes a pattern which allows a single IO-Item to subscribe to multiple topics. Received messages are forwarded to all IO-Items whose topics match the topic of the message.
5 Next, go to the Kafka Item Type section.
6 For Client Type select 'Consumer'.
7 Click Create to close the wizard and create the IO-Item object.
Sending Messages to a Kafka Topic
To send messages to a Kafka topic, IO-Item Kafka objects are used.
1 To create a new IO-Item, right-click on the Kafka Datasource object in the I/O Model and select to open the Create Object wizard.
2 In the Available Objects list, select External Interfaces > IO-Item Kafka.
3 In the Property Compounds list, select Common compound and enter an Object Name for the new IO-Item.
4 For the Item ID property provide the name of the Kafka topic.
5 Next, go to the Kafka Item Type section.
6 For the Topic property provide the name of the Kafka topic.
7 Enter the Kafka Partition if this IO-Item has to send messages to a specific partition only.
8 If a Kafka Key is to be included in messages sent via this IO_item, provide it in the Key property.
9 For Client Type select 'Producer'.
10 Click Create to close the wizard and create the IO-Item object.
Advanced Topics
Lua API
The logic for processing Kafka messages can be customized by using Lua scripts. See the Lua API.