Another benefit of the monitoring dashboards is their alerting capability. You can configure alerts based on a wide range of data and set up notifications for many types of system issues or milestones. When an event with a specific ID or severity occurs, the system can send you an email with the corresponding alert message.
For example, you can configure an alert to trigger when a storage node’s average CPU usage exceeds 85% for more than 5 minutes. The alert can include labels such as severity=critical and component=storage-node and send an email notification to system administrators with details about the affected component and current usage.
To learn more about alerts, refer to the Grafana alerts documentation.
Notification components
Notification components include three parts: alert rules, contact points, and notification policies. Together, these components determine when an alert triggers, where the notification is sent, and how it is routed.
- To set up notifications, you first create alert rules that define the conditions for triggering an alert. In general, you add one or more queries to monitor system data, set a condition that evaluates the query results, and specify additional options such as evaluation intervals and annotations that describe the alert.
-
Next, you configure contact points to decide where notifications will be delivered. Typically, you provide an email address or another destination and ensure that your system is configured to send notifications to that location.
-
Finally, you create notification policies to route alerts to your contact points. Policies let you group alerts using labels or other criteria and control which alerts go to which destinations. You can also adjust policies to handle different severity levels or types of system events.
Creating alert rules
You can add alert rules to further define triggers and control which contact points receive notifications based on specific conditions. When creating an alert rule, you can also refine the labels used in your notification policies and set the intervals at which alerts are sent.
To create an alert rule, navigate to Alerting > Alert rules and click on New alert rule.
To learn more about alert rules, refer to the Grafana alerts documentation.
Creating contact points
A contact point determines where alert notifications are sent. To create a contact point that delivers email notifications, you must enable SMTP in the monitoring service’s values.yaml configuration file. This involves adding or updating entries for the mail server address, port, authentication method, and sender information.
You can find this file in the following directory:
/kubernetes/external/grafana/helm
Then, from within Grafana, navigate to Alerting > Contact points > Add contact point and fill in the details for your point of contact.
To learn more about contact points, refer to the Grafana contact point documentation.
Creating notification policies
A notification policy controls how alerts are delivered to your contact points. By using labels, you can manage alerts and associate them with specific conditions or destinations.
Navigate to Alerting > Notification policies > New nested policy, then fill in the policy details. Create a label for the policy and select the contact point you created from the Contact point dropdown.
To learn more about notification policies, refer to the Grafana notification policies documentation. Additionally, for notification policy templates, refer to the Grafana templates documentation.
Relevant events IDs
For a list of relevent event IDs for which notifications can be set, consult the in-product VSP One Object Administration Guide.