Kafka topic schemas
Overview
A Kafka topic provides a destination for the storage of data (messages). Messages in Kafka are always sent to or received from a topic.
Producers are those client applications that publish (write) messages to Kafka, and consumers are those that subscribe to (read and process) these messages. Data contracts can be defined between the producers and the consumers, using schemas. This ensures that messages written to Kafka can always be read properly again, and prevents malformed or corrupt messages from being written.
A producer client application uses a serializer to put the messages that it sends to a specific broker topic into the correct data format. A consumer client application uses a deserializer to get the messages that it consumes from a specific broker topic into the correct data format. The schemas can be stored in Service Registry to serialize and deserialize messages, which are referenced from the client applications to ensure that the messages that they send and receive are compatible with those schemas.
Each Kafka message is a key-value pair. Therefore, we have two types of schema - key schema and value schema. A key schema enforces a structure for keys in messages sent to topics. A value schema enforces a structure for values in messages sent to topics.
Information architecture
When viewing the details for a Kafka instance, it's possible to see the key and value schemas for a specific Service Registry instance under the Schemas tab, if they are found (they are found by matching the artifact ID of the schema which follows a specific naming convention <name of the topic>-key or <name of the topic>-value).
For each found schema, the user can:
- View the schema content
- Edit the schema content in API Designer (this is a seamless editing experience by integrating API Designer’s editor into the Kafka user flow)
- View the details of that schema in the Service Registry (via a kebab menu link)
For a schema that isn’t found, the user can:
- Create a schema (using API Designer’s editor)
- Import a schema from a URL or file (using API Designer’s editor)
Examples
Empty state
The following empty state is displayed when no Service Registry instance has been selected from the dropdown.
Instance selection
The instance selection dropdown groups instances into Available instances and Unavailable instances. This grouping is only present if the list displays six or seven instances. If the list includes more instances, the grouping won’t be displayed since that would require more individual API calls which can make the rendering of the list slow.
The unavailable instances will show a tooltip upon hovering on that instance, which will display the reason for unavailability and what the user can do to access that instance.
Users can create a new Service Registry instance using the link in the dropdown’s sticky footer.
This will make the new instance available in the instance selection dropdown.
After an instance has been selected, we display a link button next to the dropdown allowing users to view instances in Service Registry.
Matching schema not found
If there are no matching key and/or value schemas found, we will display an empty state message along with action steps a user can take to create or import the schema.
Both key and value schemas are not found
One schema is not found
Edit schema
Schema can be edited in API Designer and saved as a new version in Service Registry (this is saved as a new version since each registered schema version is immutable in Service Registry).
API Designer’s editor will be surfaced here to allow the user to edit the schema seamlessly.
The user can save the edited schema as a new version in Service Registry and choose to use the auto-generated version or specify a different version.
The user can then view the schema in Service Registry or return to the Kafka Schemas tab for the previously selected Service Registry instance.
Import schema from URL
If a matching key or value schema is not found, the user can include the schema by specifying the provided artifact ID value when uploading the schema to Service Registry. Alternatively, the user can click on Create Schema or Import Schema buttons provided in the Schemas tab and the correct ID is automatically added when they save to Service Registry.
The schema will be saved to Service Registry. The user can choose to edit the schema further and save it as a new version to Service Registry via the editor.
