Understanding dynamic thresholds

An alert is generated when a threshold is exceeded. You can define a static threshold or dynamic threshold. To define a static threshold, you need to identify the workload. Because storage systems serve multiple applications and each application is managed by a different team, it becomes difficult for a storage administrator to identify an appropriate threshold for a resource.

Dynamic thresholds help you to address this challenge. Analyzer detail view utilizes historical data to determine a dynamic threshold by comparing the current values of the metric. It uses "Day Moving Average (DMA)" and “Standard Deviation (SD) for the last 7 days to determine the dynamic threshold by comparing dynamic threshold against the current day's values for the selected metric.

For a normal data set, the values within 1 SD of the mean account for about 68% of the data set, while within 2 SDs änd 3 SDs of the mean account for about 95% and 99.7% , respectively. So if you select 1 SD, then 32 % of the data points can be higher as 1 SD covers 68% data set. Similarly, for 2 SD and 3 SD thresholds, 5 % and 0.3 % data points can be higher, respectively. Set the threshold carefully based on your requirement and define the appropriate (m/n) criteria in the Criteria field. For example, if you define the criteria as 1 sample out of 30 minutes, an alert will be generated for each violation and you might receive too many alert notifications. Instead, choose the criteria for significant number of data samples for a duration. Suppose data granularity is one minute and you set the criteria as 25 samples out of 30 minutes. So, at least 30 data points out of 30 minutes are checked and an alert is generated when 25 data points exceed the threshold. It helps you to understand that something is not normal for a significant duration (in this case it is 25 out of 30 minutes).

DMA and SD values are calculated based on the historical data. For example, the DMA and SD values for storage IOPS for the last 7 days are 8000 and 2000, respectively. In this case, the dynamic threshold values for 7 DMA + 1 SD, 7 DMA + 2 SD and 7 DMA + 3 SD will be 10000, 12000 and 14000, respectively. Here, the 1 SD threshold value (10000) is 25 % higher than the 7 DMA value (8000). And 2 SD and 3 SD values are 50 % and 75 % higher than the 7 DMA value. So, you can use the thresholds 1 SD and 2 SD for the alert violation in normal cases. If you want to identify a suspicious activity (like, a malware attack), you can use the 7 DMA + 3 SD as a threshold in which current data points are continuously violating the 7 DMA + 3 SD threshold for a significant time.

The dynamic threshold value is determined everyday when a day is changed (as per Analyzer detail view application time zone).

Currently, dynamic thresholds only support a few Hitachi Block Storage configuration and performance metrics. Refer to Hitachi Analyzer Detail View Metrics Reference Guide for the list of supported metrics.