Comparing new data with old data I have a use case where I need to check if new data received this week, is different from older data received last week. The data is continuous (basically voltage & power readings from a machine).
I had used Anova to do this but that doesn't really suit my application, I was wondering if there was a statistical method to check if my new data sits within the distribution of my old data - for example when using ANOVA I can see in fact my data does sit in the old data distribution but in terms of ANOVA and resulting Pvalue - it says I have a significant difference (PVal<0.05). In reality however I really only care if the new data is 'well' outside the historic bounds..?
So the picture shows the old data (blue & orange) and the new data (green) this is statistically different in terms of Pvalue...but in reality the new data sits within the old data distributions - I'd like to capture that programmatically.
I could just literally test that fact but I just wanted to know if a more statistical approach is available..?
Paul.
 A: This is a process monitoring problem and the solution basically amounts to setting up a control chart. There are many options depending on how complicated you want to get, but the general idea is so establish a baseline value with upper and lower limits. The process is deemed to be in control when the data fall within these bounds.
In a simple case, assuming a normally distributed process, the baseline is just the mean of data that is known to have been in control, and the limits are a few (e.g., 3) standard deviations away from the mean. If the new data consistently falls outside these historic limits, then you can conclude that there is a change. How you define "consistently" is problem-specific, however.
Some issues to be aware of:


*

*Track the appropriate statistic. This is where knowledge of the system is important. That is, do you want to track the absolute measured value or the variance (scatter) or both simultaneously? For example, a power spike might be normal but rapid fluctuations between low and high might be abnormal even though the values are within the limits.

*If you want to track multiple correlated variables, tracking each one individually may not be appropriate. For example, values for each variable may be well within individual limits but far outside the joint distribution. This can be remedied with multivariate control charts.

*Define what acceptable and unacceptable process conditions are so you can be confident in the control chart - in other words, tune the control chart for a balance between true and false alarms. Historical data can be very helpful here. For example, a normally-distributed process with 3-sigma limits will throw a false alarm on average every 371 observations.


I highly recommend reading through Section 6.3 in the NIST Handbook for more details and a practical overview of how to set up control charts. Good luck!
