Alerts

Another one of the benefits to Grafana are its alerts. Based upon a wide variety of data, you can create event notifications from these alerts for a seemingly endless array of system issues or milestones.

When a system event with a specific ID or severity is triggered, Grafana will then email you with the alert text in the event messaging. This is accomplished by grouping the query results by the term message.keyword.

message.keyword maintains the event message, and can be included in the alert description or summary by using the following expression:

{{index $labels "message.keyword"}}

When creating a new alert, it is recommended that you copy one of the included example to best get started. In order for for the alert and subsequent notification to function correctly, you must then replace the provided expression above to match your desired result.

Severity filtered alert notification

To filter based on severity, use the following query string when creating an alert:

severity.keyword:desired_severity

You can then modify the default text to enter one of the following three severity levels:

INFO
SEVERE
WARNING

For example:

severity.keyword:SEVERE

Or:

severity.keyword:WARNING OR severity.keyword:SEVERE

To learn more about alerts, refer to the Grafana alerts documentation.

ID filtered alert notification

Each HCI event is represented by a unique identifier, which can also act as an alert. To filter based on a specific identifier, use the following query string when creating an alert:

identifier:REPLACE_THIS_WITH_DESIRED_IDENTIFIER

The query must be modified to include the specific event ID, which can be found in the Identifier column of the Events dashboard.

Notification components

There are three components that comprise notifications: alerts, contact points, and notification policies.

Alerts specify the criteria required for triggering the actual alert. They encompass one or multiple queries, the alert condition, evaluation behavior, annotations, and supplementary configurations.

Contact points determine the method of user notification. A typical application involves establishing a contact point via the user's email address to facilitate email notifications upon alert activation.

Lastly, notification policies are used to route alerts to the contact points you've created.

Creating an alert

The recommended approach for configuring an alert for HCI is to replicate and customize one of the provided example alerts outlined in the previous section.

Creating a contact point

A contact point determins where the alert notifications will be sent. To create a contact point, modification of the Grafana configuration is necessary to enable SMTP and configure the appropriate SMTP server. This adjustment can be made through the Admin panel found within Workflow Designer. For more information, see Making system configuration changes.

Then, from within Grafana, navigate to Alerting > Contact points > Add contact point and fill in the details for your point of contact.

To learn more about contact points, refer to the Grafana alerts documentation.

Creating the notification policy

A notification policy determines how alerts are sent to your various contact points. Using labels, you can modify the

Within Grafana, navigate to Alerting > Notification policies > New nested policy and fill in the details of your policy, making sure to create a label for the policy and select the contact point you just created from the Contact point dropdown.

To learn more about notification policies, refer to the Grafana notification policies documentation.

Creating alert rules

Alert rules can be added to your alerts to further define their triggers and refine which contact points receive them based upon the specific conditions. While creating an alert rule, you can also further define the labels create for your notification policies.

To learn more about alert rules, refer to the Grafana alerts documentation.