# Detect abnormal sample among a group of samples

Edited:

The example I used seems to be improper. Hope the following better explains my question.

I have observations of an experiment in which a product is tested many times. So there are many groups of data, not with equal size due to some missing values, like:

test1       test2     test3       test4        ...
1  -1.50371845  0.64130233 0.8376865  0.07750849        ...
2  -0.12187922 -1.90071432 0.6648617 -0.65444761        ...
3  -0.57726711  0.77819843 0.5192241 -0.57657857        ...
4  -1.07764739  1.91085958 0.6094460 -0.64624790        ...
5   0.62637053 -0.55543142 0.1513395 -0.96672391        ...
6   1.13612121  0.10154322 0.5553948 -0.20668588        ...
7  -1.40391833  0.07758314 0.1479182 -0.79954503        ...
8   0.29265407  0.47484095 0.7293415  0.64495836        ...
9   0.09265496 -2.18251767 0.6086569 -1.84081178        ...
10  0.83082910  0.76895884 0.9689856  1.01996433        ...
11 -0.48054893 -0.24780135 0.2642277  0.95435584        ...
12          NA  0.77400592 0.8213820  0.95938743        ...
13          NA -0.45984539 0.6763886          NA        ...

These groups of data are supposed to follow some pattern if nothing went wrong. My question is how to detect the abnormal groups/columns of data.

As suggested by @Navin Manaswi , I can use correlation, so the abnormal run is the one not linearly related to others. But one concern of using correlation is when there are outliers that produce a high correlation coefficient, e.g.

Also the sample size is not fixed for every run.

• This is not a question about programming, it is a question about statistics. You should ask such questions over at Cross Validated, not Stack Overflow. Apr 3, 2016 at 5:43
• Yes, you're right. And thanks for migrating it to cross validated. @MrFlick Apr 3, 2016 at 17:21

library(dplyr)
library(reshape2)
set.seed(10)

ts <- data.frame(x = 4*cos(0.2*(1:20)), y = 4*cos(0.2*(1:20))+ runif(20) ,z=  4*sin(0.2*(1:20)) + runif(20), w = 1:20, q= 3.5*cos(0.2*(1:20))+ runif(20))
cor_ts <- as.matrix(cor(df))
cor_ts_ord <- arrange(melt(cor_ts), abs(value)) # ascending order in terms of absolute value
final <- cor_ts_ord %>% group_by(Var1) %>% summarise(median = median(abs(value))) %>% arrange(median) %>% data.frame()
final[1,1] # final[1:2,1]if we want to identify two most abnormal ones
• Re the post notice: unexplained code is little better than a one-line answer. What is this code intended to do? What are its inputs and outputs? How should one interpret the output? How could somebody make use of your answer in some other programming environment?
– whuber
Apr 3, 2016 at 15:06
• Do you mean the anomaly is the one with largest median of correlations with others? @Navin Manaswi Apr 4, 2016 at 2:22