Kafka consumer groups
Author: Jenn Giardino | Last edit: May 01, 2023 | Design type: Side drawer, Tables | Product area: Kafka
Overview
A Kafka consumer is a client application that subscribes to (reads and processes) events.
Consumer groups share large data streams generated by producers from a given topic in a Kafka instance. Grouping consumers scales consumption to keep up with the rate of data produced. Consumers within a group don’t read data from the same partition, but can read data from one or more partitions.
The following examples illustrate how a user can view all the consumer groups that have access to a particular Kafka instance, as well as reset the offsets of consumer groups or delete consumer groups.
Information architecture
When viewing the details for a Kafka instance, it's possible to see a list of all consumer groups for all topics in a Kafka instance, or drill down into a specific topic to see a list of consumer groups for that topic only. The UIs for viewing consumer group details are similar except for the following cases:
- Viewing consumer groups for a specific topic
- The selected topic is presented in the page header
- When viewing the consumer group details drawer, property labels reenforce that a specific topic is selected by ending with “in this topic”
- Viewing all consumer groups for a Kafka instance
- When viewing the consumer group details drawer, the table includes a Topic column to indicate topic within which the partition exists.
Examples
Empty state
Table view
The following example shows how a drawer can display to provide more details about the selected consumer group. This drawer is accessible with the table row click or by selecting the View partition offsets action in the kebab menu.
Drawer contents
When viewing consumer groups at the Kafka instance level
When viewing consumer groups at the Kafka topic level
Partitions with lag help popover
Actions menu
Resetting the offset
A reset changes the offset position from which consumers read from the message log of a topic partition.
When the user first selects the action Reset offset for a consumer group, a modal opens, prompting the user to select which topic to manage.
After selecting a topic, the user sees options for specifying the new offset followed by a table listing all partitions in the topic and their current offset values.
If the user selects Custom offset, the user can specify a specific offset in the message log to be applied to all selected topics.
The user can also select Earliest or Latest in the New offset menu. Earliest resets to the earliest offset at the start of the message log. Latest resets to the latest offset at the end of the message log.
Before the user can apply the new offset value, the user must acknowledge the risk associated with resetting the offset.
To reset an offset position, the consumer group must have NO MEMBERS connected to a topic. If the user selects the action Reset offset for a consumer group that still has active members, then an error alert would display in the modal.
Deleting a consumer group
To view examples of the delete modal for a consumer group, refer to https://www.uxd-hub.com/entries/entry/confirmation-prompts-for-deleting-resources
Concepts and terminology
The following was copied from RHOSAK documentation and explain concepts illustrated in the examples above.
Consumer groups
Consumer Group ID
The consumer group ID is the unique identifier for the consumer group within the cluster. This is part of the consumer configuration for the application client.
Active Members
Active members shows the number of consumers in the group that are assigned to a topic partition in the Kafka instance. If you’re viewing information about consumer groups for a topic, these are the active members for the topic.
Partitions with lag
Partitions with lag shows the number of partitions where the assigned consumer has not caught up with the last message in the partition. The lag reflects the position of the consumer offset in relation to the end of the partition log.
Consumer offset positions
When you click a consumer group, you see the details of each member.
Partition
The partition number for the topic.
Client ID
The unique ID of the client application used to identify active consumers. If no client ID is shown, the partition is not currently being consumed.
Current offset
The current offset number for the consumer in the partition log. This is the position of the consumer in the partition and the latest read position.
Log end offset
The current offset number for the producer in the partition log. This is the end of the log and the latest write position.
Offset lag
The difference (delta) between the consumer and producer offset positions in the log.
Consumer lag indicators
Consumer lag for a given consumer group indicates the delay between the last message added in a partition and the message currently being picked up by that consumer. The lag reflects the position of the consumer offset in relation to the end of the partition log.
When you’re reviewing consumer group properties in the OpenShift Streams for Apache Kafka web console, look for the differences between Current offset and Log end offset. The difference shows as the Offset lag value.
For applications that rely on the processing of (near) real-time data, it’s critical that consumer lag doesn’t become too big. Suppose a topic streams 100 messages per second. A lag of 1000 messages between the producer offset (the topic partition head) and the last (current) offset that the consumer has read means a 10-second delay. The offset lag shows that a gap is opening up between the write and read positions.
To reduce lag, you typically add new consumers to a group. However, you can also increase the retention time for a message to remain in a topic. Extending the retention of data in the log gives the consumer a chance to catch up before data is flushed from the message log.