Available metrics

Content Platform for Cloud Scale Administration Guide

Version
2.6.x
File Size
1945 KB
Audience
anonymous
Part Number
MK-HCPCS008-10

Metrics provide information about the operation of a service. Metrics are collected while the service is active. If a service restarts, its metrics are restarted.

The metrics described here fall into these categories:

  • Counter - A numeric value that can only increase or be reset to zero. A counter tracks the number of times a specific event has occurred. An example is the number of S3 servlet operations.
  • Gauge - A counter that can increase or decrease. An example of a gauge is the number of active connections.
  • Histogram - A set of grouped samples. A histogram approximates the distribution of numerical data.
Note: If a metric is measured over an interval (for example, http_s3_servlet_requests_latency_seconds), but doesn't have at least two data points, the value is reported as NaN.
Note: Policy Engine activity can cause a lag in the collection of metrics.

Metrics from all services

The following metrics are available from all services.

Metric Description
http_healthcheck_requests_total The total number of requests made to the health verification API.
http_monitoring_requests_total The total number of requests made to the monitoring API.
scheduled_policy_work_items The total number of work items processed by each scheduled policy.

A work item is defined as:

  • DELETE_BACKEND_OBJECTS - Each StoredObjectID in thesu system
  • DELETE_EXPIRED_OBJECTS - Each client object, expired or not, in the system
  • DELETE_FAILED_WRITES - Each client object in the system that is in the state OPEN
  • DELETE_INCOMPLETE_MULTIPARTS - Each in-progress multi-part entry in the system
  • STORAGE_COMPONENT_HEALTH_CHECKS - Each storage component in the system
scrape_duration_seconds The duration in seconds of the scrape (collection interval).
scrape_samples_post_metric_relabeling The count of samples remaining after metric relabeling was applied.
scrape_samples_scraped The count of samples the target exposed.
up 1 if the instance is healthy (reachable) or 0 if collection of metrics from the instance failed.

Data Lifecycle

The following metrics are available from the Data Lifecycle service. Metrics are recorded for the following policies. Not every metric applies to every policy.

  • CHARGEBACK_POPULATION
  • CLIENT_OBJECT_POLICY
  • DELETE_BACKEND_OBJECTS
  • EXPIRE_FAILED_WRITE
  • INCOMPLETE_MPU_EXPIRATION
  • TOMBSTONE_DELETION
  • VERSION_EXPIRATION
Note: As of v2.4, the policies VERSION_EXPIRATION and EXPIRE_FAILED_WRITE display only historical data for the metrics lifecycle_policy_concurrency and lifecycle_policy_list_latency_seconds.
Metric Description
lifecycle_policy_accept_latency_seconds The lifecycle policy acceptance processing latency in seconds.
lifecycle_policy_completed The total number of lifecycle policies completed.
lifecycle_policy_concurrency The total number of threads currently running for the policy.
lifecycle_policy_conflicts The total number of lifecycle policy conflicts.
lifecycle_policy_deleted_backend_objects_​count The total number of objects deleted from backend storage by the policy DELETE_BACKEND_OBJECTS.
lifecycle_policy_errors The total number of errors that occurred while executing lifecycle policy actions, in the categories:
  • General
  • Listing
  • Metadata
  • S3
lifecycle_policy_examine_latency_seconds The lifecycle policy examination processing latency in seconds.
lifecycle_policy_expiration_completed_count The total number of objects completely processed by the expiration policies DELETE_MARKER and PERMANENT_DELETE).
lifecycle_policy_list_latency_seconds The lifecycle policy listing latency in seconds.
lifecycle_policy_rekey_initiated_count The number of times a rekey operation has been initiated.
lifecycle_policy_rekeyed_objects_count The total number of objects rekeyed.
lifecycle_policy_splits The total number of lifecycle policy splits.
lifecycle_policy_started The total number of lifecycle policies started.
lifecycle_policy_submitted The total number of lifecycle policies submitted.
s3_operation_count The count of S3 operations (READ, WRITE, DELETE, and HEAD) per storage component.
s3_operation_error_count The count of failed S3 operations (READ, WRITE, DELETE, and HEAD) per storage component.
s3_operation_latency_seconds The latency of storage component operations (READ, WRITE, DELETE, and HEAD) in seconds.

Key Management Server

The following metrics are available from the Key Management Server service. These metrics are collected every five minutes.

Metric Description
kmip-servers_offline The count of KMS servers that are offline. Updated hourly.
kmip_servers_online The count of KMS servers that are online. Updated hourly.
kmip_total_kek_count The count of key encryption keys stored in the KMS server. This count increments when an HCP S Series Node is added or when a rekey occurs.
lifecycle_policy_rekey_initiated_count The count of how many times rekeying has been initiated through either the MAPI method or the Object Storage Management application.
lifecycle_policy_rekeyed_objects_count The total count of data encryption keys that are re-wrapped with key encryption keys.

MAPI Gateway

The following metrics are available from the MAPI Gateway service. These metrics are collected every five minutes.

Metric Description
storage_available_capacity_bytes The number of bytes free on an HCP S Series Node.
storage_total_capacity_bytes The number of bytes total, available and used, on an HCP S Series Node.
storage_total_objects The number of objects on an HCP S Series Node.
storage_used_capacity_bytes The number of bytes used on an HCP S Series Node.

Each metric is reported with a label, store, identifying it as being either from a specific HCP S Series Node or the aggregate total. You can also retrieve the metrics using this label. For example, to retrieve the used storage capacity of the storage component hcps10.company.com, you would specify:

storage_used_capacity_bytes{store="hcps10.company.com"}

To retrieve the number of objects stored on the HCP S Series Node storage component snode67.company.com, you would specify:

storage_total_objects{instance="hcpcs_cluster:9992",job=MAPI-Gateway",store="snode67.company.com"}

To retrieve the used storage capacity of all available storage components, you would specify:

storage_used_capacity_bytes{store="aggregate"}
Note: If storage components other than HCP S Series Nodes are configured, aggregate totals aren't reported.

Message Queue

The Message Queue service supports a large number of general metrics. Information on these metrics is available at https://github.com/rabbitmq/rabbitmq-prometheus/blob/master/metrics.md.

Metadata Coordination

The following metrics are available from the Metadata Coordination service.

Metric Description
mcs_copies_per_partition Gauge of the number of copies of each metadata partition per key space (to verify protection). Two copies means available but not fault tolerant; three copies means available and fault tolerant.
mcs_disk_usage_per_instance Gauge of the total disk usage of each metadata instance.
mcs_disk_usage_per_partition Gauge of the disk usage of each metadata partition per key space.
mcs_failed_moves_per_keyspace Counter of the number of unsuccessful requests for metadata partition moves per keyspace.
mcs_failed_splits_per_keyspace Counter of the number of unsuccessful requests for metadata partition splits per keyspace.
mcs_moves_per_keyspace Counter of the number of successful requests for metadata partition moves per keyspace.
mcs_partitions_per_instance Gauge of the total number of metadata partitions per metadata instance. This is useful to verify balance and determine when scaling might be necessary.
mcs_splits_per_keyspace Counter of the number of successful requests for metadata partition splits per keyspace.

Metadata Gateway

The following metrics are available from the Metadata Gateway service.

Note:
  1. Client count metrics are an approximation and might not correspond to the actual count.
  2. Depending on when garbage collection tasks run, the ratio of client objects size to stored objects size might show a discrepancy.
Metric Description
async_action_count The count of actions performed.
async_action_latency_seconds_bucket A histogram for the duration, in seconds, of actions on buckets. For actions comprising multiple steps, this is the total of all steps.
async_action_latency_seconds_count The count of action latency measurements taken.
async_action_latency_seconds_sum The sum of action latency in seconds.
async_concurrency A gauge for the number of concurrent actions.
async_duq_latency_seconds_bucket A histogram for the duration, in seconds, of operations on the durable update queue.
async_duq_latency_seconds_count The count of durable update queue latency measurements.
async_getwork_database_count The number of driver work checks accessing the database.
async_getwork_optimized_count The number of driver work checks avoiding the database.
async_duq_latency_seconds_sum The sum of actions on durable update queue in seconds.
metadata_available_capacity_bytes The free bytes per instance (node) for the Metadata Gateway service. The label store is either the instance or aggregate.

Note: Because multiple service instances can run on a node, all consuming the same shared disk space, the value returned by this metric might be more than the actual capacity available.

metadata_clientobject_active_count The count of client objects in metadata that are in the ACTIVE state.
metadata_clientobject_active_encrypted_​count The count of encrypted client objects in metadata that are in the ACTIVE state.
metadata_clientobject_active_​unencrypted_​count The count of unencrypted client objects in metadata that are in the ACTIVE state.
metadata_clientobject_and_part_active_​space the space occupied by client objects and parts in metadata that are in the ACTIVE state.
metadata_clientobject_part_active_count The count of client object parts in metadata that are in the ACTIVE state.
metadata_storedObject_active_space The space occupied by stored objects on the back-end storage components.
metadata_used_capacity_bytes The used bytes per instance (node) for the Metadata Gateway service. The label store gives the domain name of the instance.

Note: Because multiple service instances can run on a node, all consuming the same shared disk space, combining this value with the value of metadata_available_capacity_​bytes won't give the total capacity of the service.

update_queue_inprogress The count of update queue entries in progress.
update_queue_size The size of the update queue.

Mirror In

The following metrics are available from the Mirror In service.

Metric Description
mirror_failed_total The count of failed mirror operations, both whole objects and multipart uploads.

The mirror (synchronization) type is IN.

mirror_mpu_bytes The number of bytes synchronized as part of multi-part uploads (using MultiPartUpload). This metric is updated as uploads proceed.

The mirror (synchronization) type is OUT.

mirror_mpu_errors The count of multi-part upload synchronization errors.

The mirror (synchronization) type is IN.

The client types are:

  • EXTERNAL_S3 - External metadata or storage
  • HCPCS - HCP for cloud scale metadata or storage component
  • TRANSFER - Policy Engine service

The error categories are:

  • AUTHENTICATION - unable to mirror sue to invalid credentials or permissions
  • METADATA - failure connecting to Metadata Gateway service
  • OPERATION_ABORTED - mirror operation canceled (MPU was canceled by external party)
  • RESOURCE_NOT_FOUND - object not found
  • S3 - failure to transfer data between source and target
  • SERVICE_UNAVAILABLE - service not available at time of request
  • GENERAL - uncategorized error
mirror_mpu_objects The count of objects synchronized using multi-part uploads (using MultiPartUpload).

The mirror (synchronization) type is IN.

mirror_skipped The count of skipped mirror operations, on both whole objects and multi-part uploads.

The mirror (synchronization) type is IN.

mirror_success_total The count of objects successfully synchronized.

The mirror (synchronization) type is IN.

mirror_whole_bytes_total The number of bytes synchronized as whole objects (using PutObject).

The mirror (synchronization) type is IN.

mirror_whole_errors_total The count of non-multipart synchronization errors (using PutObject).

The mirror (synchronization) type is IN.

The client types are:

  • EXTERNAL_S3 - External metadata or storage
  • HCPCS - HCP for cloud scale metadata or storage component
  • TRANSFER - Policy Engine service

The error categories are:

  • AUTHENTICATION - unable to mirror sue to invalid credentials or permissions
  • METADATA - failure connecting to Metadata Gateway service
  • OPERATION_ABORTED - mirror operation canceled (MPU was canceled by external party)
  • RESOURCE_NOT_FOUND - object not found
  • S3 - failure to transfer data between source and target
  • SERVICE_UNAVAILABLE - service not available at time of request
  • GENERAL - uncategorized error
mirror_whole_objects_total The count of objects synchronized as whole objects (using PutObject).

The mirror (synchronization) type is IN.

s3_operation_count_total The count of S3 operations (READ, WRITE, DELETE, and HEAD) per storage component

The mirror (synchronization) type is IN.

sync_from_bytes_copied The number of bytes synchronized by full copy from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_bytes_putcopied The number of bytes synchronized by put-copy from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_object_count_failed The count of objects that failed to synchronize from external storage (sync-from) by this instance, grouped by class of error. The error classes are AUTHENTICATION, METADATA, OPERATION_ABORTED, RESOURCE_NOT_FOUND, S3, SERVICE_UNAVAILABLE, and UNKNOWN.
sync_from_object_count_succeeded The count of objects synchronized from external storage (sync-from) by this instance.
sync_from_object_size_total Total size of object data synchronized from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_objects Total number of objects synchronized from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.

Mirror Out

The following metrics are available from the Mirror Out service.

Metric Description
mirror_failed_total The count of failed mirror operations, both whole objects and multi-part uploads.

The mirror (synchronization) type is OUT.

mirror_mpu_bytes The number of bytes synchronized as part of multi-part uploads (using MultiPartUpload). This metric is updated as uploads proceed.

The mirror (synchronization) type is OUT.

mirror_mpu_errors The count of multi-part upload synchronization errors.

The mirror (synchronization) type is OUT.

The client types are:

  • EXTERNAL_S3 - External metadata or storage
  • HCPCS - HCP for cloud scale metadata or storage component
  • TRANSFER - Policy Engine service

The error categories are:

  • AUTHENTICATION - unable to mirror sue to invalid credentials or permissions
  • METADATA - failure connecting to Metadata Gateway service
  • OPERATION_ABORTED - mirror operation canceled (MPU was canceled by external party)
  • RESOURCE_NOT_FOUND - object not found
  • S3 - failure to transfer data between source and target
  • SERVICE_UNAVAILABLE - service not available at time of request
  • GENERAL - uncategorized error
mirror_mpu_objects The count of objects synchronized using multi-part uploads (using MultiPartUpload).

The mirror (synchronization) type is OUT.

mirror_skipped The count of skipped mirror operations, on both whole objects and multi-part uploads.

The mirror (synchronization) type is OUT.

mirror_success_total The count of objects successfully synchronized.

The mirror (synchronization) type is OUT.

mirror_whole_bytes_total The number of bytes synchronized as whole objects (using PutObject).

The mirror (synchronization) type is OUT.

mirror_whole_errors_total The count of non-multipart synchronization errors (using PutObject).

The mirror (synchronization) type is OUT.

The client types are:

  • EXTERNAL_S3 - External metadata or storage
  • HCPCS - HCP for cloud scale metadata or storage component
  • TRANSFER - Policy Engine service

The error categories are:

  • AUTHENTICATION - unable to mirror sue to invalid credentials or permissions
  • METADATA - failure connecting to Metadata Gateway service
  • OPERATION_ABORTED - mirror operation canceled (MPU was canceled by external party)
  • RESOURCE_NOT_FOUND - object not found
  • S3 - failure to transfer data between source and target
  • SERVICE_UNAVAILABLE - service not available at time of request
  • GENERAL - uncategorized error
mirror_whole_objects_total The count of objects synchronized as whole objects (using PutObject).

The mirror (synchronization) type is OUT.

s3_operation_count_total The count of S3 operations (READ, WRITE, DELETE, and HEAD) per storage component

The mirror (synchronization) type is IN.

sync_to_bytes_copied The number of bytes synchronized by full copy to external storage (sync-to) by this instance. This metric is updated as synchronization proceeds.
sync_to_bytes_putcopied The number of bytes synchronized by put-copy (previously copied) to external storage (sync-to) by this instance.
sync_to_objects The count of objects synchronized to external storage (sync-to) by this instance.
sync_to_object_size_total The total size of object data synchronized to external storage (sync-to) by this instance. This metric is updated as synchronization proceeds.

Policy Engine

The following metrics are available from the Policy Engine service.

Metric Description
confirm_latency_seconds_created The message queue publish confirmation latency in seconds.
duq_query_latency The time to get a response from a get_duq query.
duq_query_latency_count The number of times the durable update queue (DUQ) has been queried (for determining the average).
duq_query_latency_sum The aggregate sum of latencies for DUQ queries (for determining the average).
mq_all_bucket_lookup_latency_seconds Average latency from a lookup of all buckets.
mq_all_mirror_count_total The count of messages dispatched to mirror exchange.
mq_all_mirror_drop_count_total The count of messages filtered from mirror exchange.
mq_all_notification_count_total The count of messages dispatched to notification exchange.
mq_all_notification_drop_count_total The count of messages filtered from notification exchange.
mq_queued_messages

Gauge of the queue depth (number of messages) that are being processed, or waiting to be processed, in these product queues:

  • s3.all - Messages resulting from all S3 operations.
  • s3.mirroringEvents - Messages for objects that require mirroring out to an external bucket. This is the Sync-To backlog.
  • s3.mirrorTransfer - Messages for objects that require mirroring in from an external bucket. This is the Sync-From backlog. Note: This queue is limited to 1 million entries. If the queue fills reading from SQS pauses. There might be additional backlog in SQS.
  • s3.notificationEvents - Messages for objects that require S3 notification to external entities. This is the S3 External Notification backlog.
  • lifecycle.chargeback - Messages that define tasks for aggregating chargeback data. This is the chargeback lifecycle policy task backlog.
  • lifecycle.delete-backend - Messages that define tasks for reclaiming space from storage components. This is the delete backend object lifecycle policy task backlog.
  • lifecycle.expire-mpu - Messages that define tasks for expiring multipart uploads. Expiration is defined using a bucket lifecycle policy. This is the expire in-progress MPU lifecycle policy task backlog.
  • lifecycle.client-object-policy - Messages that define tasks for client object policies, including version expiration, delete marker expiration, and tombstone expiration. Expiration for versions and delete markers are defined using a bucket lifecycle policy. This is the client object expiration lifecycle policy task backlog.
  • lifecycle.mirror-table-maintenance - This is the mirror tracking table lifecycle policy task backlog.

Note: A task represents a range of objects. Each range can have many thousands of objects.

policy_engine_errors_total Count of how many errors per error type per instance.
policy_engine_operations_total Count of how many time a policy ran per policy type per instance (similar to http_s3_servlet_operations_total). Operations include both asynchronous and scheduled operations, such as sync_to, sync_from, and sched_storage_component_healthchecks_examined.
policy_engine_time_total Total time spent processing requests per instance. This helps measure load balancing between instances of the Policy Engine service.
sync_from_bytes The number of bytes synchronized from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_bytes_copied The number of bytes synchronized by full copy from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_bytes_putcopied The number of bytes synchronized by put-copy from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_from_objects Total number of objects synchronized from external storage (sync-from) by this instance. This metric is updated as synchronization proceeds.
sync_to_bytes The number of bytes synchronized to external storage (sync-to) by this instance. This metric is updated as synchronization proceeds.
sync_to_bytes_copied The number of bytes synchronized by full copy to external storage (sync-to) by this instance. This metric is updated as synchronization proceeds.
sync_to_bytes_putcopied The number of bytes synchronized by put-copy (previously copied) to external storage (sync-to) by this instance.
sync_to_object_count_failed The count of objects that failed to synchronize to external storage (sync-to) by this instance, grouped by class of error. The error classes are AUTHENTICATION, METADATA, OPERATION_ABORTED, RESOURCE_NOT_FOUND, S3, SERVICE_UNAVAILABLE, and UNKNOWN.
sync_to_object_count_succeeded The count of objects synchronized to external storage (sync-to) by this instance.
sync_to_objects The count of objects synchronized to external storage (sync-to) by this instance.

RabbitMQ

RabbitMQ is a third-party application that is used by HCP for cloud scale to coordinate tasks submitted to the Policy Engine service for asynchronous processing. You can log in to the RabbitMQ interface to observe queue health. The following metrics are available from RabbitMQ:

  • The number of messages in the queue
  • The number of confirmed messages
  • The number of unconfirmed (unacknowledged) messages
  • The number of consumed (delivered and acknowledged) messages
  • The number of unroutable returned messages
  • The number of nodes in the RabbitMQ cluster

S3 Gateway

The following metrics are available from the S3 Gateway service.

Metric Description
http_s3_monitoring_requests_created The timestamp when the counter http_s3_monitoring_requests_total was created.
http_s3_monitoring_requests_total The total count of S3 monitoring requests.
http_s3_servlet_errors_total The total number of errors returned by the s3 servlet, grouped by error.
http_s3_servlet_get_object_response_​bytes_created The timestamp when the counter http_s3_servlet_get_object_response_​bytes_total was created.
http_s3_servlet_get_object_response_​bytes_per_bucket_created The timestamp when the counter http_s3_servlet_get_object_response_​bytes_per_bucket_total was created.
http_s3_servlet_get_object_response_​bytes_per_bucket_total The total number of total bytes in the body of S3 GET object responses per bucket.
http_s3_servlet_get_object_response_​bytes_total The total number of bytes in the body of S3 GET object responses.
http_s3_servlet_ingest_object_bytes_per_​bucket_created The timestamp when the counter http_s3_servlet_ingest_object_bytes_per_​bucket_total was created.
http_s3_servlet_ingest_object_bytes_per_​bucket_total The total count of objects ingested for the specified bucket.
http_s3_servlet_operations_created The timestamp when the counter http_s3_servlet_operations_total was created.
http_s3_servlet_operations_total The total number of S3 operations made to the s3 servlet for each method, grouped by operation.
http_s3_servlet_post_object_bytes_created The timestamp when the counter http_s3_servlet_post_object_bytes_total was created.
http_s3_servlet_post_object_bytes_total The total number of bytes of objects posted to S3.
http_s3_servlet_put_copied_bytes_total The number of total bytes of objects PUT copied (previously copied) to S3.
http_s3_servlet_put_object_bytes_created The timestamp when the counter http_s3_servlet_put_object_bytes_total was created.
http_s3_servlet_put_object_bytes_total The number of total bytes of objects PUT (previously copied) to S3.
http_s3_servlet_put_object_part_bytes_​total The number of total bytes of PUT part operations (previously copied) to S3.
http_s3_servlet_requests_histogram_​latency_seconds The latency in seconds as measured by a histogram timer, grouped by operation.
http_s3_servlet_requests_histogram_​latency_​seconds_bucket The latency in seconds as measured by a histogram timer, grouped by bucket.
http_s3_servlet_requests_histogram_​latency_​seconds_count The count of s3 servlet request observations; used with sum to determine average.
http_s3_servlet_requests_histogram_​latency_​seconds_sum Sum of s3 servlet request latency in seconds; used with count to determine average.
http_s3_servlet_requests_latency_seconds The latency in seconds as measured by a summary timer, grouped by operation.
http_s3_servlet_requests_latency_seconds:hour_average The latency in seconds over the last hour as measured by a summary timer.
http_s3_servlet_requests_latency_seconds_count  
http_s3_servlet_requests_latency_seconds_sum The sum of request latency in seconds.
http_s3_servlet_requests_per_bucket_​created The timestamp when the counter http_s3_servlet_requests_per_bucket_total was created.
http_s3_servlet_requests_per_bucket_total The total count of total put, get, or deletion requests made to the specified bucket.
http_s3_servlet_requests_created The timestamp when the counter http_s3_servlet_requests_total was created.
http_s3_servlet_requests_total The total number of requests made to the s3 servlet, grouped by method.
http_s3_servlet_unimplemented_api_​request_created The timestamp when the counter http_s3_servlet_unimplemented_api_​request_total was created.
http_s3_servlet_unimplemented_api_​request_total The total number of requests made for unimplemented S3 methods.
http_s3_servlet_unimplemented_bucket_​api_​request_total The total number of requests made for unimplemented S3 methods per bucket, grouped by API.
http_s3_servlet_unimplemented_object_​api_request_total The total number of requests made for unimplemented S3 methods per object, grouped by API.
http_s3_servlet_unimplemented_service_​api_request_total The total number of requests made for unimplemented S3 methods per service, grouped by API.
http_s3_servlet_unknown_api_requests_​total The total number of requests made for unknown S3 methods, grouped by API.
s3_operation_error_count The count of failed S3 operations (READ, WRITE, DELETE, and HEAD) per storage component
s3_operation_latency_seconds The latency of storage component operations (READ, WRITE, DELETE, and HEAD) in seconds
s3select_total_bytes_scanned The number of bytes scanned in the object
s3select_total_bytes_processed The number of bytes processed by the request
s3select_total_bytes_returned The number of bytes returned from the request
s3select_input_type Count of requests by file type
s3select_output_type Count of responses by file type

S3 Notification

The following metrics are available from the S3 Notification service.

Metric Description
mq_publish_latency_seconds The message queue publishing latency in seconds.
notification_events_considered_total The count of events considered that could lead to notifications.
notification_events_notification_attempted_​total The count of events that had at least one notification message attempted.
notification_message_failures_total The count of notification messages that were attempted but failed.
notification_message_parsing_failures_total The count of candidate object events that could not be parsed.
notification_messages_sent_total The count of notification messages that were successfully sent.
notification_message_target_generation_​failures_total The count of candidate objects for which a list of notification targets could not be generated.