{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Observability Solution from CloudWatch: Apache Kafka on EC2. Version: 1.0.0.", "Transform": "AWS::LanguageExtensions", "Parameters": { "DashboardNameParameter": { "Type": "String", "Description": "Enter name of the dashboard, the maximum length is 255, and valid characters are A-Z, a-z, 0-9, \"-\", and \"_\". The dashboard will display metrics from the region where the stack is being created. To easily differentiate this dashboard from similar ones in other regions, we recommend including the region name in the dashboard name.", "MinLength": "1", "MaxLength": "255", "AllowedPattern": "^[a-zA-Z0-9-_]*$", "Default": "KafkaDashboard" } }, "Resources": { "CloudWatchDashboard": { "Type": "AWS::CloudWatch::Dashboard", "Properties": { "DashboardName": { "Ref": "DashboardNameParameter" }, "DashboardBody": { "Fn::ToJsonString": { "variables": [ { "type": "pattern", "pattern": "KafkaClusterName", "inputType": "select", "id": "KafkaClusterName", "label": "Kafka Cluster", "defaultValue": "__FIRST", "visible": true, "search": "{CWAgent,ClusterName,InstanceId} MetricName=kafka.leader.election.rate", "populateFrom": "ClusterName" }, { "type": "pattern", "pattern": "KafkaProducerGroupName", "inputType": "select", "id": "KafkaProducerGroupName", "label": "Kafka Producer", "defaultValue": "__FIRST", "visible": true, "search": "{CWAgent,ProducerGroupName,InstanceId,client-id} MetricName=kafka.producer.request-rate", "populateFrom": "ProducerGroupName" }, { "type": "pattern", "pattern": "KafkaConsumerGroupName", "inputType": "select", "id": "KafkaConsumerGroupName", "label": "Kafka Consumer Group", "defaultValue": "__FIRST", "visible": true, "search": "{CWAgent,ConsumerGroupName,InstanceId,client-id} MetricName=kafka.consumer.fetch-rate", "populateFrom": "ConsumerGroupName" } ], "widgets": [ { "height": 5, "width": 8, "y": 7, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.leader.election.rate\"', 'Sum', 60)))", "id": "e1", "period": 60, "label": "Leader Election Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Leader election rate across cluster" } }, { "height": 3, "width": 24, "y": 0, "x": 0, "type": "text", "properties": { "markdown": "# Cluster overview\nThis part of the dashboard gives you a high-level summary and overall picture of the activities and performance happening on your selected Kafka cluster. A Kafka cluster can consist of a single broker or multiple brokers working together. To view metrics by broker, see the Brokers section of the dashboard.\n\n\n**Note:** To customize the metrics displayed in this section, select different cluster names from the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 4, "width": 8, "y": 3, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(SEARCH('{CWAgent,ClusterName,InstanceId} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.partition.under_replicated\"', 'Average', 60))", "label": "Partitions (Count)" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false } }, "sparkline": true, "view": "singleValue", "region": { "Ref": "AWS::Region" }, "title": "Partitions under replicated", "period": 60, "stat": "Average" } }, { "height": 4, "width": 8, "y": 3, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(SEARCH('{CWAgent,ClusterName,InstanceId} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.partition.offline\"', 'Average', 60))", "label": "Partitions (Count)" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false } }, "sparkline": true, "view": "singleValue", "region": { "Ref": "AWS::Region" }, "title": "Offline partitions", "period": 60, "stat": "Average" } }, { "height": 4, "width": 8, "y": 3, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "expand - shrink", "label": "Delta", "id": "delta" } ], [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,operation,InstanceId} ClusterName=\"KafkaClusterName\" operation=\"expand\" MetricName=\"kafka.isr.operation.count\"', 'Sum', 60)))", "id": "expand", "visible": false } ], [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,operation,InstanceId} ClusterName=\"KafkaClusterName\" operation=\"shrink\" MetricName=\"kafka.isr.operation.count\"', 'Sum', 60)))", "id": "shrink", "visible": false } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false } }, "sparkline": true, "view": "singleValue", "region": { "Ref": "AWS::Region" }, "title": "In-sync replicas (ISR) delta", "period": 60, "stat": "Average" } }, { "height": 6, "width": 8, "y": 59, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,state,InstanceId} ClusterName=\"KafkaClusterName\" state=\"in\" MetricName=\"kafka.network.io\"', 'Sum', 60)), MAX, DESC, 10)", "id": "in", "period": 60, "label": "In [avg: ${AVG}, max: ${MAX}]" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,state,InstanceId} ClusterName=\"KafkaClusterName\" state=\"in\" MetricName=\"kafka.network.io\"', 'Sum', 60)))", "id": "in1", "period": 60, "label": "Average In [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Network in throughput by instance" } }, { "height": 6, "width": 8, "y": 65, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,state,InstanceId} ClusterName=\"KafkaClusterName\" state=\"out\" MetricName=\"kafka.network.io\"', 'Sum', 60)), MAX, DESC, 10)", "id": "out", "period": 60, "label": "Out [avg: ${AVG}, max: ${MAX}]" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,state,InstanceId} ClusterName=\"KafkaClusterName\" state=\"out\" MetricName=\"kafka.network.io\"', 'Sum', 60)))", "id": "out1", "period": 60, "label": "Average Out [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Network out throughput by instance" } }, { "height": 6, "width": 8, "y": 59, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,type,InstanceId} ClusterName=\"KafkaClusterName\" type=\"fetch\" MetricName=\"kafka.purgatory.size\"', 'Average', 60)), MAX, DESC, 10)", "id": "in", "period": 60, "label": "Fetch Requests [avg: ${AVG}, max: ${MAX}]" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,type,InstanceId} ClusterName=\"KafkaClusterName\" type=\"fetch\" MetricName=\"kafka.purgatory.size\"', 'Average', 60)))", "id": "in1", "period": 60, "label": "Average Fetch Requests [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Fetch requests purgatory size by instance" } }, { "height": 6, "width": 8, "y": 47, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchfollower\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60)), MAX, DESC, 10)", "label": "Request Time [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}", "id": "m1" } ], [ { "expression": "AVG(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchfollower\" MetricName=\"kafka.request.time.avg\"', 'Average', 60))", "label": "Average Request Time [avg: ${AVG}, max: ${MAX}]", "id": "m2" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Follower request time" } }, { "height": 6, "width": 8, "y": 74, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id, topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.record-error-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "error_rate", "period": 60, "label": "Error Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.topic')}, ${PROP('Dim.InstanceId')}" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Record error rate by topic and instanceId" } }, { "height": 6, "width": 8, "y": 80, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "request_rate", "period": 60, "label": "Request Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-rate\"', 'Average', 60)))", "id": "request_rate1", "period": 60, "label": "Average Request Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Request rate" } }, { "height": 6, "width": 8, "y": 74, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-latency-avg\"', 'Average', 60)), MAX, DESC, 10)", "id": "request_rate", "period": 60, "label": "Latency [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-latency-avg\"', 'Average', 60)))", "id": "request_rate11", "period": 60, "label": "Average Latency [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Request latency" } }, { "height": 6, "width": 8, "y": 95, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id,topic} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.bytes-consumed-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "e1", "period": 60, "label": "Byte Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.topic')}, ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id,topic} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.bytes-consumed-rate\"', 'Average', 60)))", "id": "e12", "period": 60, "label": "Average Byte Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Bytes consumption rate by topic and instanceId" } }, { "height": 6, "width": 8, "y": 89, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id,topic} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-consumed-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "e1", "period": 60, "label": "Records Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.topic')}, ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id,topic} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-consumed-rate\"', 'Average', 60)))", "id": "e21", "period": 60, "label": "Average Records Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Messages consumed by topic and instanceId" } }, { "height": 6, "width": 8, "y": 89, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.fetch-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "e1", "period": 60, "label": "Fetch Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.client-id')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.fetch-rate\"', 'Average', 60)))", "id": "e12", "period": 60, "label": "Average Fetch Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Message fetch rate" } }, { "height": 6, "width": 8, "y": 53, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.leader.election.rate\"', 'Sum', 60)))", "id": "e1", "period": 60, "label": "Leader Election Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Leader election rate" } }, { "height": 3, "width": 24, "y": 17, "x": 0, "type": "text", "properties": { "markdown": "# Producer group overview\nThis part of the dashboard gives you a high-level summary and overall picture of the activities and performance happening on your selected Kafka prodcuer group. A Kafka producer group can consist of a single producer or multiple producers working together. To view metrics by producer, see the Producers section of the dashboard.\n\n\n**Note**: To view the metrics of a different producer group in this section, select that producer group in the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 3, "width": 24, "y": 27, "x": 0, "type": "text", "properties": { "markdown": "# Consumer group overview\nThis part of the dashboard gives you a high-level summary and overall picture of the activities and performance happening on your selected Kafka consumer group. A Kafka consumer group can consist of a single consumer or multiple consumers working together. To checkout metrics by consumer, please refer to consumers section below.\n\n\n**Note**: To see the metrics of a different consumer group in this section, select that consumer group name in the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 3, "width": 24, "y": 44, "x": 0, "type": "text", "properties": { "markdown": "# Brokers\nThe metrics displayed on this part of the dashboard provide insights into potential data loss or delays that could occur due to unclean leader elections in the Kafka cluster or network throughput issues. Additionally, it shows request failure information based on requests getting stuck in request purgatory states or hitting timeout thresholds.\n\n\n**Note**: To view the metrics of different clusters in this section, select that cluster name in the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 3, "width": 24, "y": 71, "x": 0, "type": "text", "properties": { "markdown": "# Producers\nThe Kafka producers send data messages to the brokers, where the messages are stored and distributed across different topics. The metrics shown in this section give you information about the amount and speed of data being sent by the producers, as well as the rate at which these messages are successfully delivered to the brokers.\n\n\n**Note**: To view the metrics of a different producer group in this section, select that producer group in the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 3, "width": 24, "y": 86, "x": 0, "type": "text", "properties": { "markdown": "# Consumers\nThe Kafka consumer metrics provide visibility into any delays or lags between when messages are produced and when they are actually consumed. Additionally, these metrics show the success rates for message delivery to consumers, the latency involved in this process, and the total volume or amount of data being consumed.\n\n\n**Note**: To view the metrics of a different consumer group in this section, select that consumer group in the dropdown list at the top of the dashboard.", "background": "transparent" } }, { "height": 7, "width": 8, "y": 20, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-rate\"', 'Average', 60)))", "label": "Average Request Rate [avg: ${AVG}, max: ${MAX}]", "id": "request_rate" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.response-rate\"', 'Average', 60)))", "label": "Average Response Rate [avg: ${AVG}, max: ${MAX}]", "id": "response_rate" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Average request/response rate" } }, { "height": 7, "width": 8, "y": 20, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.request-latency-avg\"', 'Average', 60)))", "label": "Average Request Latency [avg: ${AVG}, max: ${MAX}]", "id": "request_latency" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Average request latency" } }, { "height": 7, "width": 8, "y": 20, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id, topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.record-send-rate\"', 'Average', 60)))", "label": "Average Record Send Rate [avg: ${AVG}, max: ${MAX}]", "id": "record_send_rate" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id, topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.record-error-rate\"', 'Average', 60)))", "label": "Average Record Error Rate [avg: ${AVG}, max: ${MAX}]", "id": "record_error_rate" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Average record send/error rate" } }, { "height": 7, "width": 8, "y": 30, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.fetch-rate\"', 'Average', 60)))", "label": "Average Consumer Fetch Rate [avg: ${AVG}, max: ${MAX}]", "id": "fetch_rate" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Average consumer fetch rate" } }, { "height": 7, "width": 8, "y": 30, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id,topic} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-consumed-rate\"', 'Average', 60)))", "label": "Total Messages Consumed [avg: ${AVG}, max: ${MAX}]", "id": "messaged_consumed" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Average message consumption rate" } }, { "height": 5, "width": 8, "y": 12, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.count\" type=\"fetch\"', 'Sum', 60)))", "label": "Total Fetch Requests [avg: ${AVG}, max: ${MAX}]", "id": "total_fetch_requests", "visible": false } ], [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.failed\" type=\"fetch\"', 'Sum', 60)))", "label": "Total Failed Fetch Requests [avg: ${AVG}, max: ${MAX}]", "id": "total_failed_fetch_requests", "visible": false } ], [ { "expression": "(total_failed_fetch_requests/total_fetch_requests)*100", "label": "Fetch Failure Percent", "id": "fetch_failure_percent", "color": "#d62728" } ] ], "view": "timeSeries", "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Fetch failure percentage", "yAxis": { "left": { "label": "Percent", "showUnits": false, "max": 100, "min": 0 } }, "stacked": false } }, { "height": 5, "width": 8, "y": 12, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.count\" type=\"produce\"', 'Sum', 60)))", "label": "Total Fetch Requests [avg: ${AVG}, max: ${MAX}]", "id": "total_produce_requests", "visible": false } ], [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.failed\" type=\"produce\"', 'Sum', 60)))", "label": "Total Failed Fetch Requests [avg: ${AVG}, max: ${MAX}]", "id": "total_failed_produce_requests", "visible": false } ], [ { "expression": "(total_failed_produce_requests/total_produce_requests)*100", "label": "Produce Failure Percent", "id": "produce_failure_percent", "color": "#d62728" } ] ], "view": "timeSeries", "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Produce failure percentage", "yAxis": { "left": { "label": "Percent", "showUnits": false, "min": 0, "max": 100 } }, "stacked": false } }, { "height": 6, "width": 8, "y": 53, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(failed_requests, MAX, DESC, 10)", "label": "Failed Fetch Requests [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}", "id": "e1" } ], [ { "expression": "AVG(failed_requests)", "label": "Average Failed Fetch Requests [avg: ${AVG}, max: ${MAX}]", "id": "e2" } ], [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.failed\" type=\"fetch\"', 'Sum', 60)), MAX, DESC, 10)", "label": "Failed Consumer Requests", "id": "failed_requests", "visible": false } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Failed fetch requests " } }, { "height": 6, "width": 8, "y": 53, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(failed_requests, MAX, DESC, 10)", "label": "Failed Producer Requests [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}", "id": "e1" } ], [ { "expression": "AVG(failed_requests)", "label": "Average Failed Producer Requests [avg: ${AVG}, max: ${MAX}]", "id": "e2" } ], [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ClusterName,type} ClusterName=\"KafkaClusterName\" MetricName=\"kafka.request.failed\" type=\"produce\"', 'Sum', 60)), MAX, DESC, 10)", "label": "Failed Producer Requests", "id": "failed_requests", "visible": false } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Failed producer requests " } }, { "height": 5, "width": 8, "y": 7, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "MAX(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60))", "label": "Maximum Producer Request Time [avg: ${AVG}, max: ${MAX}]", "id": "m1" } ], [ { "expression": "AVG(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.request.time.avg\"', 'Average', 60))", "label": "Average Producer Request Time [avg: ${AVG}, max: ${MAX}]", "id": "m2" } ] ], "view": "timeSeries", "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Maximum producer request time", "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "stacked": false } }, { "height": 5, "width": 8, "y": 7, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "MAX(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchconsumer\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60))", "label": "Maximum Consumer Fetch Time [avg: ${AVG}, max: ${MAX}]", "id": "m1" } ], [ { "expression": "AVG(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchconsumer\" MetricName=\"kafka.request.time.avg\"', 'Average', 60))", "label": "Average Consumer Fetch Time [avg: ${AVG}, max: ${MAX}]", "id": "m2" } ] ], "view": "timeSeries", "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Maximum consumer fetch time", "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "stacked": false } }, { "height": 6, "width": 8, "y": 74, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id, topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.record-send-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "send_rate", "period": 60, "label": "Send Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.topic')}, ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id, topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.record-send-rate\"', 'Average', 60)))", "id": "send_rate1", "period": 60, "label": "Average Send Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Record send rate by topic and instanceId" } }, { "height": 7, "width": 8, "y": 30, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SUM(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-lag-max\"', 'Maximum', 60)))", "label": "Consumer Lag [avg: ${AVG}, max: ${MAX}]", "id": "comsumer_lag" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Consumer lag" } }, { "height": 6, "width": 8, "y": 89, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-lag-max\"', 'Average', 60)), MAX, DESC, 10)", "id": "e1", "period": 60, "label": "Consumer Lag [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.client-id')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.records-lag-max\"', 'Average', 60)))", "id": "e2", "period": 60, "label": "Average Consumer Lag [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Consumer lag" } }, { "height": 6, "width": 8, "y": 47, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchconsumer\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60)), MAX, DESC, 10)", "label": "Request Time [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}", "id": "m1" } ], [ { "expression": "AVG(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"fetchconsumer\" MetricName=\"kafka.request.time.avg\"', 'Average', 60))", "label": "Average Request Time [avg: ${AVG}, max: ${MAX}]", "id": "m2" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Consumer request time" } }, { "height": 6, "width": 8, "y": 47, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60)), MAX, DESC, 10)", "label": "Request Time [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}", "id": "m1" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,InstanceId,type} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.request.time.avg\"', 'Maximum', 60)))", "label": "Average Request Time [avg: ${AVG}, max: ${MAX}]", "id": "m2" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Producer request time" } }, { "height": 6, "width": 8, "y": 59, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,type,InstanceId} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.purgatory.size\"', 'Maximum', 60)), MAX, DESC, 10)", "id": "in", "period": 60, "label": "Produce Requests [avg: ${AVG}, max: ${MAX}]" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,ClusterName,type,InstanceId} ClusterName=\"KafkaClusterName\" type=\"produce\" MetricName=\"kafka.purgatory.size\"', 'Maximum', 60)))", "id": "in1", "period": 60, "label": "Average Produce Requests [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Produce requests purgatory size" } }, { "height": 7, "width": 8, "y": 37, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ConsumerGroupName,client-id} ConsumerGroupName=\"KafkaConsumerGroupName\" MetricName=\"kafka.consumer.total.bytes-consumed-rate\"', 'Average', 60)), MAX, DESC, 10)", "label": "Bytes Consumed Rate [avg: ${AVG}, max: ${MAX}]", "id": "bytes_consumed_rate" } ] ], "yAxis": { "left": { "label": "Bytes/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Bytes consumption rate" } }, { "height": 6, "width": 8, "y": 80, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id,topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.byte-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "byte_rate", "period": 60, "label": "Byte Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.topic')}, ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id,topic} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.byte-rate\"', 'Average', 60)))", "id": "byte_rate1", "period": 60, "label": "Average Byte Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Byte Rate by topic and instanceId" } }, { "height": 6, "width": 8, "y": 80, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.response-rate\"', 'Average', 60)), MAX, DESC, 10)", "id": "response_rate", "period": 60, "label": "Response Rate [avg: ${AVG}, max: ${MAX}] ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(REMOVE_EMPTY(SEARCH('{CWAgent,InstanceId,ProducerGroupName,client-id} ProducerGroupName=\"KafkaProducerGroupName\" MetricName=\"kafka.producer.response-rate\"', 'Average', 60)))", "id": "response_rate1", "period": 60, "label": "Average Response Rate [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count/Second", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "period": 60, "stat": "Average", "title": "Response rate" } }, { "height": 2, "width": 24, "y": 101, "x": 0, "type": "text", "properties": { "markdown": "# JVM host metrics\nThe following sections provide a more detailed look at the top contributing servers for various JVM metrics, in your selected Kafka Cluster.", "background": "transparent" } }, { "height": 8, "width": 8, "y": 105, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.used\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Used [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.used\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Used [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 heap memory used" } }, { "height": 8, "width": 12, "y": 123, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.threads.count\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Threads Count [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.threads.count\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Threads Count [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 threads count" } }, { "height": 8, "width": 12, "y": 123, "x": 12, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.classes.loaded\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Classes Loaded [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.classes.loaded\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Classes Loaded [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 classes loaded" } }, { "height": 8, "width": 8, "y": 113, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.used\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Used [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.used\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Used [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 non-heap memory used" } }, { "height": 2, "width": 24, "y": 103, "x": 0, "type": "text", "properties": { "markdown": "## Memory usage by host\nThis section provides visibility into the memory usage of hosts in the selected JVM process group. Heap usage metrics help you identify memory leaks and manage memory allocation issues. The non-heap metrics are crucial for monitoring the performance of the JVM.\nWhen maximum heap memory or non-heap memory displays as negative, it means that the maximum is not configured and the metric emits -1 for that metric.", "background": "transparent" } }, { "height": 2, "width": 24, "y": 121, "x": 0, "type": "text", "properties": { "markdown": "## Threads and classes loaded by host\nThis section provides visibility into the threads and classes loaded of hosts on the selected JVM process group. An unexpected high number of threads can indicate issues such as thread leaks or high concurrency demands. The number of classes loaded can help detect excessive dynamic class creation.", "background": "transparent" } }, { "height": 3, "width": 24, "y": 131, "x": 0, "type": "text", "properties": { "markdown": "## Garbage collection\nThis section provides visibility into the garbage collection of hosts on the selected JVM process group. Use the number of garbage collections to monitor memory issues or suboptimal JVM settings. High garbage collection time might indicate that the JVM is struggling to free memory efficiently.\nThe heap is partitioned into a set of equal-sized heap regions, each a contiguous range of virtual memory with no fixed size for Eden, Survivor, or Old. This provides greater flexibility and efficiency. G1 (Garbage first) performs a concurrent global marking phase to determine the liveness of objects throughout the heap.\nMinor GC moves the live objects from Eden to Survivor 1 (or Survivor 2) when Eden memory exceeds its limit. In a mixed garbage collection, the G1 GC optionally adds some old regions to the set of eden and survivor regions that will be collected. The exact number of old regions added is controlled by a number of flags. Full GC performs in-place compaction of the entire heap and might be slow. Full garbage collections are still single threaded, but if tuned properly your applications should avoid full garbage collections.\n", "background": "transparent" } }, { "height": 8, "width": 12, "y": 144, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.count\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Young Generation\"', 'Sum', 60)", "id": "e2", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e2, MAX, DESC, 10)", "label": "Invocations Per Minute [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(e2)", "label": "Average Invocations [${AVG}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection invocations per minute" } }, { "height": 1, "width": 24, "y": 134, "x": 0, "type": "text", "properties": { "markdown": "### G1 concurrent garbage collection invocations and duration", "background": "transparent" } }, { "height": 8, "width": 12, "y": 144, "x": 12, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.elapsed\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Young Generation\"', 'Sum', 60)", "id": "e1", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e1, MAX, DESC, 10)", "label": "Duration [last: ${LAST}, max: ${MAX}] - ${PROP('Dim.InstanceId')}", "id": "m1" } ], [ { "expression": "AVG(e1)", "label": "Avg Duration [${AVG}]", "id": "m3" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection duration" } }, { "height": 1, "width": 24, "y": 143, "x": 0, "type": "text", "properties": { "markdown": "### Minor garbage collection invocations and duration", "background": "transparent" } }, { "height": 8, "width": 12, "y": 135, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.count\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Concurrent GC\"', 'Sum', 60)", "id": "e2", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e2, MAX, DESC, 10)", "label": "Invocations Per Minute [last: ${LAST}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(e2)", "label": "Average Invocations [${AVG}]", "id": "m2" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection invocations per Minute" } }, { "height": 8, "width": 12, "y": 135, "x": 12, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.elapsed\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Concurrent GC\"', 'Sum', 60)", "id": "e2", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e2, MAX, DESC, 10)", "label": "Duration [last: ${LAST}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(e2)", "label": "Average Duration [${AVG}]", "id": "m2" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection duration" } }, { "height": 1, "width": 24, "y": 152, "x": 0, "type": "text", "properties": { "markdown": "### Mixed garbage collection invocations and duration", "background": "transparent" } }, { "height": 8, "width": 12, "y": 153, "x": 0, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.count\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Old Generation\"', 'Sum', 60)", "id": "e2", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e2, MAX, DESC, 10)", "label": "Invocations Per Minute [last: ${LAST}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(e2)", "label": "Average Invocations [${AVG}]" } ] ], "yAxis": { "left": { "label": "Count", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection invocations per minute" } }, { "height": 8, "width": 12, "y": 153, "x": 12, "type": "metric", "properties": { "metrics": [ [ { "expression": "SEARCH('{CWAgent,InstanceId,ProcessGroupName,name} MetricName=\"jvm.gc.collections.elapsed\" ProcessGroupName=\"KafkaClusterName\" name=\"G1 Old Generation\"', 'Sum', 60)", "id": "e2", "period": 60, "label": "Expression2", "visible": false } ], [ { "expression": "SORT(e2, MAX, DESC, 10)", "label": "Duration [last: ${LAST}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(e2)", "label": "Avg Duration [${AVG}]" } ] ], "yAxis": { "left": { "label": "Milliseconds", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 garbage collection duration" } }, { "height": 8, "width": 8, "y": 105, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.committed\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Committed [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.committed\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Committed [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 heap memory committed" } }, { "height": 8, "width": 8, "y": 105, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.max\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Max [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.heap.max\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Max [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 heap memory max" } }, { "height": 8, "width": 8, "y": 113, "x": 8, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.committed\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Committed [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.committed\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Committed [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false, "min": 0 } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 non-heap memory committed" } }, { "height": 8, "width": 8, "y": 113, "x": 16, "type": "metric", "properties": { "metrics": [ [ { "expression": "SORT(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.max\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60), MAX, DESC, 10)", "id": "m1", "period": 60, "label": "Max [avg: ${AVG}, max: ${MAX}] - ${PROP('Dim.InstanceId')}" } ], [ { "expression": "AVG(SEARCH('{CWAgent,InstanceId,ProcessGroupName} MetricName=\"jvm.memory.nonheap.max\" ProcessGroupName=\"KafkaClusterName\"', 'Average', 60))", "id": "e1", "period": 60, "label": "Average Max [avg: ${AVG}, max: ${MAX}]" } ] ], "yAxis": { "left": { "label": "Bytes", "showUnits": false } }, "view": "timeSeries", "stacked": false, "region": { "Ref": "AWS::Region" }, "stat": "Average", "period": 60, "title": "Top 10 non-heap memory max" } }, { "height": 3, "width": 24, "y": 161, "x": 0, "type": "text", "properties": { "markdown": { "Fn::Join": [ "", [ "# EC2 metrics\nTo view the AWS/EC2 vended metrics for your EC2 instances, refer to the [Amazon EC2 CloudWatch Automatic Dashboard](https://", { "Ref": "AWS::Region" }, ".console.aws.amazon.com/cloudwatch/home?region=", { "Ref": "AWS::Region" }, "#home:dashboards/EC2).\n\n\nTo view system-level metrics collected by the CloudWatch agent deployed to your EC2 instances, refer the Monitoring tab for the relevant instance on the [Amazon EC2 console](https://", { "Ref": "AWS::Region" }, ".console.aws.amazon.com/ec2/home#Instances)." ] ] }, "background": "transparent" } } ] } } } } } }