ConsumerCoordinator

  • Fetch thread to pull msgs.
  • Heartbeat thread to the coordinator. Default timeout at 10s (session.timeout.ms)
    • Will get the state of the coordinator, if it is rebalancing, send the JoinGroup request
  • Every consumer group has a coordinator on the broker. The coordinator manages the membership of consumers

Kafka group states

  • Empty: init state, no group member
  • PreparingBalance
  • InitialRebalance: delay entering PreparingBalance state to reduce rebalance times
  • AwaitSync: waiting for the new assignment from the group leader
  • Stable
  • Dead: empty group. Metadata no longer available

When joining new group, the state is Empty -> PreparingBalance -> AwaitSync -> Stable

Rebalance consumer

When the consumer membership changes, coordinator will rebalance consumer to partition association

  • Coordinator replys the heartbeat request from all consumers with rebalance notification. All consumers need to resend joinGroup request
  • JoinGroup: When new one joins the group
    • the coordinater will let the consumer group leader know the membership assignment, generaton
    • Other consumers will know the leader, generation and topic and its partition info
  • SyncGroup: after JoinGroup stage is over. Consumer leader will rebalance and return the assignment to the coordinator, which in turn broadcast to other consumers as reponse of SyncGroup call from followers
    • Default assignment strategy is RangeAssignor
  • If leader is lost during the rebalance, it will be detected by coordinator and the leader will be removed