We gather various metrics for a system many times per hour. Then for each hour, compute and store the mean, variance, and sample size of each metric.
Later to evaluate load and system changes, on an interval (say a day or week) we compute Welch's t test between corresponding hours of the same interval a week or month previous. If the mean of these t tests is above a threshold, we consider that a significant change has occurred in that metric.
Is Welch's t test the right choice? We use it because the mean, variance, and size may be different between the compared hours. Is there a better choice that takes into account that this time series data?
Overall, is this a reasonable approach? Is there a better way of identifying significant changes?
