Kafka Bridging Overview

Kafka bridging allows for the configuration of two objects to bridge between a Kafka cluster and an SAP software event broker. Once configured, message flow is possible in both directions. This behavior is directly embedded in the event broker, in other words, no external Kafka Connect infrastructure is required in order to pass messages to and from Kafka.

A Kafka bridge consists of the following configuration objects:

• A Kafka Receiver —receives events from one or more Kafka topics, converts the events to Solace Message Format (SMF) messages, and publishes them to topics on the SAP event broker.

• A Kafka Sender—takes SMF messages from one or more queues, converts the messages to Kafka events, and sends them to Kafka topics on the remote Kafka cluster.

Kafka Receiver

Each Kafka receiver contains a list of topic bindings. Each topic binding names the Kafka topic that messages are drawn from, and includes attributes that dictate how messages from the Kafka topic are translated into a Solace message.

Messages translated from a Kafka topic are published into the Solace message bus. Because a Kafka topic may not be appropriate as an SAP topic, you must use a substitution expression to construct an SAP topic for the received message. Substitution expressions are an SAP-specific language used to replace specific text attributes with system generated output. A common pattern in Kafka topics, for example, is to embed a hierarchy in them using dots or underscores (e.g. "a.b.c"), and an expression such as ${replace(kafkaTopic(),".","/")} would convert these to an SMF topic and allow for more rich topic matching once within the SAP network.

The following table describes how messages are converted from Kafka brokers to SAP event brokers:

Kafka	SAP	Comments
payload	payload	The payload is always written to the SAP binary attachment.
partition key	partition key	The SAP partition key is set based on how you configure the topic binding's local key substitution expression.
topic	user property k_topic
partition number	user property k_partition
partition offset	user property k_offset
timestamp	sender timestamp
headers	user properties	All Kafka headers are converted to SAP user properties.
<various>	topic	The SAP topic is set based on how you configure the topic binding's local topic substitution expression.

For information about how to set up an event broker to receive events from a Kafka cluster using a Kafka receiver, see Creating a Kafka Receiver.

Kafka Sender

Each Kafka sender contains a list of queue bindings. Each queue binding names the queue that messages are drawn from, and includes attributes that dictate how those messages are sent to a Kafka cluster. These are ordinary queues, configured outside of the context of the Kafka sender, and as such the messages that the queue attracts depend on the set of queue subscriptions configured for it. These messages are then sent to a single Kafka topic configured per queue binding.

Because Kafka topics are partitioned, with partition choice being determined by the client (in this case the SAP event broker), each Kafka sender offers several partition choice options:

• Random—select a random partition of the Kafka topic.

• Explicit—select an explicit partition of the Kafka topic.

• Consistent—use a hash of a partition key to select a partition.

When using a consistent partition selection scheme, you can use a substitution expression to determine the partition key . For example, you could use the expression ${topic()}, which would generate a partition key which matches the published Solace topic. This would result in all messages with the same Solace topic being sent to the same Kafka topic partition.

A substitution expression can be used to generate a key for the Kafka message even if a consistent partition scheme is not being used. There might be other entities in the Kafka cluster that use the key, and being able to set it using a substitution expression can be useful, even if the Kafka sender doesn't use it to determine the partition.

The following table describes how messages are converted from SAP event brokers to Kafka brokers:

Solace	Kafka	Comments
payload	payload
	topic	The Kafka topic is set based on how you configure the queue binding's remote topic attribute. This is a single fixed value per queue binding.
<various>	partition key	The Kafka partition key is generated per-message based on how you configure the queue binding's remote key substitution expression.
sender timestamp	timestamp
user properties	headers	All SAP user properties are converted to Kafka headers.

For information about how to set up the event broker to send messages to a Kafka cluster using a Kafka sender, see Creating a Kafka Sender.