0
$\begingroup$

Edited:

The example I used seems to be improper. Hope the following better explains my question.

I have observations of an experiment in which a product is tested many times. So there are many groups of data, not with equal size due to some missing values, like:

         test1       test2     test3       test4        ...
1  -1.50371845  0.64130233 0.8376865  0.07750849        ...
2  -0.12187922 -1.90071432 0.6648617 -0.65444761        ...
3  -0.57726711  0.77819843 0.5192241 -0.57657857        ...
4  -1.07764739  1.91085958 0.6094460 -0.64624790        ...
5   0.62637053 -0.55543142 0.1513395 -0.96672391        ...
6   1.13612121  0.10154322 0.5553948 -0.20668588        ...
7  -1.40391833  0.07758314 0.1479182 -0.79954503        ...
8   0.29265407  0.47484095 0.7293415  0.64495836        ...
9   0.09265496 -2.18251767 0.6086569 -1.84081178        ...
10  0.83082910  0.76895884 0.9689856  1.01996433        ...
11 -0.48054893 -0.24780135 0.2642277  0.95435584        ...
12          NA  0.77400592 0.8213820  0.95938743        ...
13          NA -0.45984539 0.6763886          NA        ...

These groups of data are supposed to follow some pattern if nothing went wrong. My question is how to detect the abnormal groups/columns of data.

As suggested by @Navin Manaswi , I can use correlation, so the abnormal run is the one not linearly related to others. But one concern of using correlation is when there are outliers that produce a high correlation coefficient, e.g. Anscombe's quartet

Also the sample size is not fixed for every run.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ This is not a question about programming, it is a question about statistics. You should ask such questions over at Cross Validated, not Stack Overflow. $\endgroup$
    – MrFlick
    Apr 3, 2016 at 5:43
  • $\begingroup$ Yes, you're right. And thanks for migrating it to cross validated. @MrFlick $\endgroup$
    – Roy C
    Apr 3, 2016 at 17:21

1 Answer 1

Reset to default
0
$\begingroup$ 
library(dplyr)
library(reshape2)
set.seed(10)

 ts <- data.frame(x = 4*cos(0.2*(1:20)), y = 4*cos(0.2*(1:20))+ runif(20) ,z=  4*sin(0.2*(1:20)) + runif(20), w = 1:20, q= 3.5*cos(0.2*(1:20))+ runif(20))
cor_ts <- as.matrix(cor(df))
cor_ts_ord <- arrange(melt(cor_ts), abs(value)) # ascending order in terms of absolute value
final <- cor_ts_ord %>% group_by(Var1) %>% summarise(median = median(abs(value))) %>% arrange(median) %>% data.frame()
final[1,1] # final[1:2,1]if we want to identify two most abnormal ones
Improve this answer
$\endgroup$
2
  • $\begingroup$ Re the post notice: unexplained code is little better than a one-line answer. What is this code intended to do? What are its inputs and outputs? How should one interpret the output? How could somebody make use of your answer in some other programming environment? $\endgroup$
    – whuber ♦
    Apr 3, 2016 at 15:06
  • $\begingroup$ Do you mean the anomaly is the one with largest median of correlations with others? @Navin Manaswi $\endgroup$
    – Roy C
    Apr 4, 2016 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.