Topic Architecture Case Studies
The following case studies show how to apply topic architecture best practices:
- Case Study 1: Event-Driven Microservices, Choreography, and Event Sourcing
- Case Study 2: Event-Driven Integration with ASAPIO
- Case Study 3: IoT Devices—Command, Control, and Analytics
- Case Study 4: Data Ingestion and Distribution
Case Study 1: Event-Driven Microservices, Choreography, and Event Sourcing
Implementing microservices breaks monolithic applications (services) into smaller, more manageable components (microservices). Making microservices event-driven makes them more efficient, performant, scalable, and robust.
Two key attributes of event-driven microservices are:
- Choreography, where microservices react to changes in their environment, in contrast to the orchestration approach, where the system relies on either tight linkages between microservices or a central orchestrator that, although it drives microservice interactions, is the main bottleneck that prevents scaling.
- Event sourcing allows microservices to function and recover without needing atomic transactions for large units of work by capturing the system’s state within events rather than storing it in databases. You can achieve consistency in your system by replaying specific events, rather than managing the state of tightly-coupled services.
Let's see how a well-defined topic architecture enables both the flow of events through this system and the flexibility to expand the data set. Originally, the system delivered beer by car, but as business grows, the company could add other products, such as Scotch whiskey, or add new delivery modes such as delivery from a drone, so they need to plan for business growth.
Building the Topic Root
In this system, customer-generated events are injected into the event mesh by the Sol-Beer WebApp. A customer places an order and then wants to be informed about updates to the status of their order, but they don't need details about logistics or inventory levels.
We can start from the event topic template, Domain/ObjectType/Verb/Version/Properties
. First, focusing on the event topic root, our application domain is Sol-Beer’s online store, so store
is appropriate as the Domain
. One of the main functions of the store is to accept orders, so order
may be an ObjectType
. Within an order, a few actions are possible, including: created
, updated
, and canceled
. Because this is the first version of the store, the version is v1
.Based on these elements, we have the following root topics:
store/order/created/v1
store/order/updated/v1
store/order/canceled/v1
Adding Topic Properties
After building a topic root, you can consider various properties for each individual Object/Action combination. Potential properties that may be useful in this application include:
Property | Description |
---|---|
|
A value based on an enumeration of possible product group types (beer, scotch, and so on). This property may be useful for actions taken on specific product groups, such as marketing actions on a certain group of products. |
|
The general region that an inventory manager is responsible for. This property may be useful for servicing multiple regions as delivery vehicles may only be interested in the orders in their region. |
|
The specific item that is ordered (dark beer, light beer, ten-year old scotch, and so on). This property may be useful for actions taken on specific products, for example a promotion on one certain product. |
{customerID} |
The customer identifier. This property may be useful for access control, because customers may only be able to place orders for their own ID. |
|
The unique order identifier. This property may be useful to allow a customer to receive updates for a specific order. |
Each new property is included when there is a routing, filtering, or access control use case. Each property should be ordered in the topic roughly by cardinality. In this example, there are more customers than there are products, and there are many more possible order IDs than there are customers. This leads to the following, complete topic:
store/order/created/v1/{productGroup}/{area}/{productID}/{customerID}/{orderID}
You can take a similar approach for the updated and canceled actions.
This topic taxonomy leads to many potential subscriber use cases. For example, the inventory management system may be set up to handle only specific regions. An inventory manager responsible for distribution in the Ottawa area would subscribe to the following topic:
store/order/created/v1/*/ottawa/>
Note the use of wildcards in the previous example indicates that the subscriber wants to receive messages with all possible values for that property. Here, the inventory manager is interested in orders for every product category (productCategory
), so they use a wildcard (*
) in the subscription to include all values.
Expanding the Event Catalog
In addition to processing orders, the Sol-Beer store processes payments and fulfills orders by local delivery. These events can be added to the catalog with similar topic roots as in the previous sample topics. For example:
store/billing/paid/v1
store/billing/nofunds/v1
store/delivery/enroute/v1
store/delivery/delivered/v1
If all events in the order, delivery, and billing process carry through the customerID
and orderID
, the process allows tracing and message replay for a single order or customer. This functionality is key for event sourcing. For example, when the billing events contain the customerID
and orderID
, you can trace the billing events back to order events. The resulting billing topics in the event catalog would be:
store/billing/paid/v1/{productGroup}/{area}/{deliveryMode}/{productID}/{customerID}/{orderID}
store/billing/nofunds/v1/{productGroup}/{area}/{deliveryMode}/{productID}/{customerID}/{orderID}
where the properties may look like:
beer/ottawa/drone/product456/cust4321/order12345
This rich topic taxonomy allows for many different use cases. For example, the inventory manager and the delivery vehicle that need to know when the billing has cleared for beer orders would subscribe to:
store/billing/enroute/v1/beer/ottawa/drone/product456/customer4321/order12345>
After the beer purchase has been authorized and the driver or drone assigned, the customer can track their delivery with live updates using delivery events. Some properties that may be helpful to include in the topic hierarchy are:
Property | Description |
---|---|
|
A value from an enumeration of vehicle type, such as a car, drone, and so on. |
|
The latitude of the delivery vehicle position. |
|
The longitude of the delivery vehicle position. |
|
The delivery vehicle identifier. |
The resulting event catalog entries would be:
store/delivery/enroute/v1/{vehicleType}/{vehicleID}/{customerID}/{orderID}/{latitude}/{longitude}
store/delivery/dispatched/v1/{vehicleType}/{vehicleID}/{customerID}/{orderID}
The event topic properties may look like this:
store/delivery/enroute/v1/car/vehicle4321/order12345/cust4321/45.421532/-75.697189
Some possible subscriptions for this event include:
-
a customer tracking the delivery of all of their orders could subscribe to:
store/delivery/enroute/v1/*/*/cust4321/>
-
A fleet manager tracking all their vehicles, regardless of type or current order, within a geographical area might subscribe to:
store/delivery/enroute/v1/*/*/*/*/45*/-75*
Flexible subscriptions allow events to be reused in new ways as necessary. For example, if new regulations required commercial drones to be tracked and reported to the city, a reporting agent could subscribe to the geographical area that it is responsible for:
store/delivery/enroute/v1/drone/*/*/*/45*/-75*
Case Study 2: Event-Driven Integration with ASAPIO
In this example, we'll event-enable SAP with ASAPIO . This is an example of event driven integration, where the data transformation capabilities of an iPaaS is combined with the real-time dynamic choreography of an event broker and event mesh. As shown in the diagram below, when a new purchase order is created in an SAP system, ASAPIO can push an event notification out to an event mesh in SAP, which streams that notification out to multiple interested applications (on-premises and in the cloud), such as applications used for billing, manufacturing and support. The value of this use case is that instead of each downstream application polling the system of record for changes, the system of record emits one event per change and all subscribing downstream applications have the ability to consume the event. This architecture reduces the workload on the system of record.
With ASAPIO, there can be different encodings (JSON, XML, etc.) of the same event. Because there can be only one event schema per event, we need to treat each encoding as a separate event. In this case, we include a HandlingInstruction
property in the event root. We also move the HandlingInstruction
property before the version property so that if just a single encoding changes, we can version just the one event. The result is the following event topic template:
Domain/ObjectType/Verb/HandlingInstruction/Version/Locality/SourceId/ObjectId
Let's see how a well-defined topic architecture helps data flow through this complex system.
In this use case, an event is emitted each time certain business objects (BO) change in the SAP system.
The events flowing from SAP to downstream applications have one of three different object types or aspects, each with progressively more information, which is controlled through ACLs. These aspects are described in the following table:
Aspect | Description |
---|---|
|
A notification that a particular business object has been created or updated. The topic and payload contain enough information for an application or user to retrieve the business object according to their SAP access rights. |
|
Contains the information from |
|
Contains the information from |
For these SAP events the topic template would be:
sap/{resourceType}/{action}/{representation}/{version}/{aspect}/{siteID}/{orgID}/{systemID}/{resourceID}
Item | Description |
---|---|
|
A literal value specifying that these are SAP-originating events. |
|
The business object type that the notification applies to. For example, |
|
A value from an enumeration of actions, for example create or update. |
|
The encoding of the event (e.g., XML, JSON). |
|
The version of the event. |
|
A value from an enumeration of object types |
|
The |
|
The organization unit that owns the business object, where GLOBAL indicates a global entity, and EMPTY indicates that there is no orgID associated with the notification |
|
The SAP instance or system ID. |
|
The business object ID. |
The event catalog entries, which are derived from the event topic root, would look like:
sap/BUS2032/create/JSON/v1
sap/BUS2032/update/
JSON/v1
To see how this event topic root is used, consider the following event:
sap/BUS2032/update/
JSON/v1
/sap-bo-data/empty/1000/ERD100/0000096852
As we read through the fields in the topic hierarchy, we see the following information:
- SAP event
- Business Object: BUS2032 (sales order)
- It's an update to a business object
- Encoded in JSON
- API version: 1
- Aspect: sap-bo-data (contains actual data)
- Site ID: none
- Organization ID: 1000
- SAP instance: ERD100
- Business Object ID: 96852
A downstream application that wants to receive all sales order updates with their data from organization 1000 could subscribe to:
sap/BUS2032/update/JSON/v1/sap-bo-data/*/1000/>
Note that this application would need the appropriate access permissions (via an administratively applied ACL) to allow it to subscribe to sap-bo-data
events from organization 1000.
A downstream application that wants to receive all sales order updates (event only) from any SAP instance, any organization, at any site, could subscribe to:
sap/BUS2032/*/JSON/v1/sap-bo-event/>
Again, this application would need to have the appropriate access permissions to allow it to receive notification events of new sales orders.
Case Study 3: IoT Devices—Command, Control, and Analytics
Internet of Things (IoT) use cases traditionally involve data movement exclusively from field sensors into backend state capture or analytics systems. The initial goals created simple systems that could horizontally scale to handle millions of devices. IoT uses cases have evolved as the numbers and complexity of the devices have evolved. Security is now mandatory, and command and control of end devices is much more common within these systems. The events coming from IoT sensors are now counted on for more than simple statistics. They now also drive changes in upstream systems. Some critical sensor events might require human interactions, and it is important that the events are accurate and valid. For example, consider an air-bag deployment event on a mobile vehicle and the consequences if these events are spoofed or ignored. There are now also many use cases where IoT sensors receive data from backend systems, for example, connected vehicles receiving weather and traffic updates.
Modern industrial IoT use cases integrate a large number of components into a unified system. To effectively organize and route data across these systems, a unified namespace (UNS) is needed. A UNS is a single source of truth for the real-time state of an enterprise. It allows all components within the enterprise to communicate bidirectionally in a standard way. An event mesh provides this functionality, and a well-defined topic hierarchy is key to standardizing communication, particularly by making data identifiable across all components in the system. For instance, an event published from an IoT device may be consumed by a number of backend systems, from data lakes to analytics engines. Event topics must be well constructed to be useful in these backend systems and prevent data lakes from turning into data swamps. In particular, a device ID should be included in the topic properties so that all events for a particular device can be monitored in real time through the event mesh and viewed historically in backend systems. Including the device ID in topics also allows you to secure the data through
Let's see how a well-defined topic architecture helps authorize and route events from remote device to backend systems, and how backend systems can control remote devices.
Device to Backend Services
Events emitted by edge IoT devices consist mainly of sensor data.
Starting with the topic event template of Domain/ObjectType/Verb/Version/Properties
, we have a topic template of
fromVehicle/{vehicleComponent}/{action}/{version}/{severity}/{username}/{sensorID}
Using fromVehicle
as the domain indicates that the information in the topic is coming from the sensors. The domain may need to include additional information if multiple vehicle generations are hosted on the same event mesh, for example, myCar/myGeneration/fromVehicle
.
Property | Description |
---|---|
fromVehicle
|
A literal value specifying that these are events from the vehicle. |
|
The name of the controllers or apps within the vehicle (e.g.,TCU and MCU). |
|
A value from an enumeration of sensor actions (e.g., reset, update, and failure). Different vehicle components may have different actions. |
|
The version number for this release of |
|
A value from an enumeration of severity (e.g., error, warning, and info). |
|
The unique sensor that produced the event. |
The event catalog entries derived from the event topic root, would look like:
fromVehicle/tcu/reset/v1
fromVehicle/tcu/update/v1
fromVehicle/tcu/failure/v1
fromVehicle/mcu/reset/v1
fromVehicle/mcu/update/v1
The properties of the topics provide more metadata for routing and filtering. For example, the location of the vehicle might affect event routing and the severity of the event might affect the number of applications interested in the event. Each topic should also include the sensor ID because it is crucial for access control restrictions to ensure that data from other sensors can't be spoofed.
Let's say a vehicle has an accident and its airbag deploys. The vehicle might emit an event with topic:
fromVehicle/tcu/update/v1/critical/vin4321/airbag
There could be an application listening to all critical severity events:
fromVehicle/*/update/v1/critical/>
In many cases, IoT devices are not in a secure location and can be subject to physical or software hacking. For this reason, the device must be authenticated on every connection, and the authenticated ID must be included and validated with each message to ensure the device only sends authorized events. This is the reason the MQTT username field is so important in this type of environment—it prevents one IoT device from pretending to be another IoT device.
For example, verifying the username would prevent a hacked vehicle with username vin4321
from sending an event as vin1234
. The following update from username vin4321
would be allowed:
fromVehicle/tcu/update/v1/critical/vin4321/airbag
However, the following update from username vin4321
would not be allowed:
fromVehicle/tcu/update/v1/critical/vin1234/airbag
Backend Services to Device
The data sent from backend applications to vehicles could consist of multicast messages to several vehicles or unicast messages to a specific vehicle. Multicast messages could be based on location, on-board application, vehicle type, or other criteria. For unicast messages, the username ensures that only the intended vehicle receives the update.
Consider a topic template of:
toVehicle/{vehicleComponent}/{action}/{version}/{location}/{clientID}/{sensorID}
Field | Description |
---|---|
toVehicle
|
A literal value specifying that these are events sent to the vehicle. |
|
A value from an enumeration of the controllers or apps, for example TCU, MCU, and so on. An MCU could act as multiplexor to downstream subcomponents. |
|
A value from an enumeration of actions (possibly dependent on the vehicle component). |
|
The version number for this release of orders. |
|
The Multicast area, group, or null. |
|
The username or device identifier used to validate whether the sender is authorized to produce the event. |
|
The identifier of the specific sub-component or app that should receive the control message. |
The event catalog entries, which are derived from the event topic root, would look like:
toVehicle/tcu/update/v1
toVehicle/mcu/update/v1
To differentiate multicast events that are available to all vehicles, an allVehicles
application domain could be used:
allVehicles/weather/update/v1
allVehicles/traffic/update/v1
The properties of the topics would provide more information for routing, filtering, and access control.
A vehicle would subscribe to all events directed specifically to it:
toVehicle/*/*/v1/*/vin4321/>
The vehicle could also subscribe to weather updates in its region:
allVehicles/weather/update/v1/ottawa/weatherPlan4321/>
And it could subscribe to traffic updates for the current road and direction:
allVehicles/traffic/update/v1/417east/trafficPlan4321/>
As previously mentioned, the username is used to access policies and filtering. For example, the vehicle may only be allowed to subscribe to toVehicle
domain updates where its VIN is included in the topic, preventing it from accessing information for other vehicles, whereas it may be allowed to consume any allVehicle
updates.
Case Study 4: Data Ingestion and Distribution
In this example, we look at data sources throughout an enterprise that write data into distributed data stores. These data sources could be published transaction logs from a database, online purchases, market data orders, and so on.
The following diagram illustrates an example of market data orders and post trade data, where the trade itself takes place in the Front Office and the post-trade activity takes place in the Middle Office. Here we can see that the client gateways produce events for the Front Office and the trading platform is the producer of events that flow to the Middle Office systems. We can also see the Middle Office is not a single-purpose consumer application, but a series of applications that each have specific tasks and data requirements.
Without a properly designed event topic architecture that allows event re-use, you might have a very expensive trading platform publishing the same event multiple times in multiple formats to satisfy each Middle Office need.
We start with the standard topic event template of Domain/ObjectType/Verb/Version/Properties
.
Let's see how a well-defined topic architecture helps data flow throughout this complex system.
Front Office
The message exchange pattern in this phase of the interaction consists of a simple command message. As shown in the diagram above, the data flow for the Front Office system, works like this:
- A Client registers an intent to buy or sell a product with a Client Gateway.
- The Client Gateway turns this request into a Client Order and sends it to a Trading Platform system.
- The Trading Platform system determines the appropriate Exchange to use, and places the Market Order.
- The Exchange fulfills the Market Order and sends confirmation back to the Trading Platform system.
- On receipt of a Fill event from the Exchange, the Trading Platform notifies the Client (via the Client Gateway) that its order has been filled.
For the Front Office, the topic template for client orders would be:
orders/{productGroup}/{action}/{version}/{market}/{source}/{product}
Item | Description |
---|---|
|
The literal value specifying the topic describes the total of all orders events. |
|
A value from an enumeration of possible product groups. |
|
A value from an enumeration of order actions (e.g., |
|
The version number for this release of |
|
General region or group of markets for which the trading platform is responsible. |
|
The customer identifier. |
|
The specific item being ordered. |
The event catalog entries, which are derived from the event topic root, would look like:
orders/equity/new/v1
orders/equity/update/v1
orders/equity/cancel/v1
orders/bond/new/v1
orders/bond/update/v1
orders/bond/cancel/v1
The event topic properties might look like:
orders/bond/cancel/v1/NA/pri1/customer1/appl
Monitoring apps could track individual customers using the following subscription:
orders/equity/*/v1/NA/custID4321/>
Actions like cancellations could be monitored to flag unusual activity using the following subscription:
orders/equity/cancel/v1/NA/>
Horizontal Scaling
A single application may not be able to keep up with large amounts of data. Horizontal scaling of consumers can be accomplished in many ways, such as nonexclusive queues or partitioned queues (see Message Distribution with Partitioned Queues. However, if the load is too much for a single event broker to handle, the data may need to be sharded across multiple event brokers in the event mesh using topic partitions.
In this use case, the properties of the topics provide more information for routing and filtering. The Smart Order Routers (SORs) could be sharded based on these properties and each process subsets of the products. This structure allows for horizontal scaling across multiple event brokers within the event mesh.
In this example, the SORs split the traffic, first by productGroup
and then at lower levels to handle shards or the product space.
The first SOR instance could have the following subscriptions:
orders/equity/*/v1/NA/*/a*/>
...
orders/equity/*/v1/NA/*/m*/>
And the second instance could have the following subscriptions:
orders/equity/*/v1/NA/*/n*/>
...
orders/equity/*/v1/NA/*/z*/>
Using Subscription Exceptions for Horizontal Scaling
Consider an expected high-volume event, such as an IPO for ACME Rideshare (ACME), where a large volume of trades on one specific symbol is expected for a limited period of time. To handle this situation, you could deploy a dedicated event broker to handle only the ACME Rideshare trade events, with an SOR instance subscribing to orders/equity/*/v1/NA/*/acme/>
.
To ensure that no ACME Rideshare events are propagated to other SOR instances, the SOR instance that handles the equities starting with "a" could add a negative subscription of !/equity/*/v1/NA/*/acme/>
, allowing for dedicated handling of ACME orders by the other SOR instance.
Middle Office
After the Trading Platform system has completed a trade, it sends an event message into the Middle Office system. The Middle Office system includes several systems that must have a consistent view of the trades that happen, but where each needs specific data to perform its tasks. Because this is a separate domain, the SORs would publish to a topic under the transaction domain. The event includes the specific Exchange the trade should occur on, for example:
transaction/equity/new/v1/NYSE/traderID1234/Order4321
The Middle Office exchange gateways would subscribe to the exchanges and pools they service, for example:
transaction/equity/new/v1/NYSE/>
The full topic taxonomy of the Middle Office system would be:
transaction/{productGroup}/{action}/{version}/{venue}/{source}/{orderID}
Item | Description |
---|---|
|
The literal value specifying the topic describes the total of all orders events. |
|
A value from an enumeration of possible product group types. |
|
A value from an enumeration of transaction actions such as |
|
The version number for this release of |
|
A value from an enumeration of the liquidity pools or exchanges where transactions can occur. |
{source}
|
The customer identifier. |
|
The identifier for the unique product order. |
In this example, you could imagine that applications like Settlements and Risk systems would be interested only in successfully completed transactions, whereas a Reporting system may require access to every transaction including canceled ones, and Trade Data Capture system might consume both every order event and every transaction event, for either the entire set of data or again sharded by product type, region, venue, or product.
For example, for the event:
transaction/equity/new/v1/NYSE/traderID1234/Order4321
The Settlements system may subscribe to:
transaction/equity/filled/v1/>
or:
transaction/equity/filled/v1/NYSE/>
While the Risk system might subscribe to the following topic to monitor the positions of a particular trader:
transaction/equity/*/v1/*/traderID1234/>