I'm trying to build a daily alert system to let me know when something unusual has happened in my analytics data that might require further investigation.
Right now I'm taking 7 weeks worth of data, creating an average for each day of the week (ie Average Conversion Rate for Monday). I then compare how much this Monday has changed from the regular Monday average in terms of standard deviations. If it's +/-2 stdevs, then it warrants investigation.
A few concerns I have :
How many weeks should I take into account? My concern is that if I take too many then I might have a trend emerging that skews my alerts. If I take too few, I might have high variance and the system might fire alerts needlessly.
Would it be better to take a rolling average for say last 7 Mondays and compare this Monday to that rather than the mean?
What would the best way be to deal with seasonality - ie bank holiday Mondays which happen occasionally but would skew averages and standard deviations.
Some websites might have little traffic per day. How should I deal with them? Would I have to take a longer time frame to calculate averages? How would I decide how many to take?