Apache Kafka Monitoring

Apache Kafka Performance Monitoring

Apache Kafka is a popular open source data streaming platform providing high-throughput data pipelines. As such, Instana has included comprehensive Kafka Monitoring in our Infrastructure and Application Monitoring solution. Instana discovers all Kafka instances present in the environment, understands the dependencies and interactions between the cluster and its servers, and automatically begins monitoring the health and performance of each instance.

To help DevOps optimize their Kafka clusters, Instana applies automation and artificial intelligence to Kafka performance and health monitoring, including predictive problem detection and AI-assisted root cause analysis. Instana reduces your need for explicit Kafka expertise within operations.

Kafka Performance and Health Monitoring

Instana monitors Kafka’s Key Performance Indicators (KPIs) at the individual component and application level for each instance. The Kafka KPIs are used for real-time health monitoring of each instance and cluster. The Service KPIs are used to understand Kafka’s performance within the overall environment.

Instana AI uses advanced statistical analysis, a set of specific Knowledge Base rules, and applied machine learning to determine the real-time health status of each instance and cluster.

Kafka Performance and Configuration Monitoring

Instana’s Kafka Monitoring sensor gathers all critical metrics to determine the health of each Kafka instance and the performance of all data pipelines on the cluster. Tracking Kafka configuration data allows Instana to monitor changes to the Kafka instances and correlate those changes to any overall performance issues.

The Kafka performance and configuration monitoring is summarized in a single dashboard that combines all Kafka monitoring information. The dashboard allows IT Ops and DevOps to see all relevant Kafka data in a single place for easy-problem solving and performance optimization.

Kafka performance monitoring centers around metrics relevant to its interactions with the data pipelines and dependant applications that live in and around the Kafka cluster. Instana’s Kafka sensor automatically identifies and collects those relevant metrics. Below is a screenshot of Instana’s Kafka performance dashboard:

The Kafka Monitoring Dashboard shows Kafka’s performance at the node, cluster, and broker levels, as well as Kafka configuration information.

Kafka Monitoring Data

Instana Kafka Monitoring includes three types of data: Tracked Configuration, Performance Metrics, and Health Signatures:

Tracked Configuration Performance Metrics Health Signatures
Version Broker Traffic Active Controller Count
Request Latency vs. Throughput isrShrinks, isrExpansions
Broker Failures Leader Elections
Broker State Metrics Unclean Leader Elections
Average Idle Time Network Processor Low Idle
Partitions Offline Partitions
Log Flushing Request Handler Low Idle
Under Replicated Partitions

Kafka Monitoring Sensor Installation: Getting Started

Ready to start monitoring Kafka? Begin by signing up for an Instana Trial or Account. If you already have an account, check out Instana’s Kafka Monitoring Documentation.

Play with Instana’s APM Observability Sandbox

Start your FREE TRIAL today!