Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a series of 100 points

My dataset can be found here . Each row is a data series. The plot for 90th row is

enter image description here

It's easy to detect outliers visually by plotting example. I tried using hampel to find outliers assuming it as time series.

x <- read.table("anomaly_s57.dat")
data <- as.matrix(x)

plot_hampel = function(row, k = 2, t = 3) {

    plot.ts(data[row,])
    hp <- hampel(data[row,], k , t)
    y <- data[row,hp$ind]
        x <- hp$ind
    points(x,y,col="red")

} 

But it is not good enough. It misses some small peaks. Is the time series assumption not correct? Any statistical fitting is possible? How to detect outliers in this data series considering each row of the data set as independent data series? It is known that total number of outliers is around 500.

share|improve this question
    
I searched for "hampel" in R and found one defunct statistic and one from the rlm (robust linear model) function in the MASS library. Can you say what you sere using? –  Peter Flom Nov 24 '12 at 13:56
    
I don't get this. Every column is an observation? –  user603 Nov 24 '12 at 14:34
    
@PeterFlom I was using hampel from MASS –  Pankaj More Nov 24 '12 at 15:16
    
@user603 Each row is an independent dataset –  Pankaj More Nov 24 '12 at 15:17
    
@Pankaj More: ok but they all have 100 columns. Are these columns related? Can we think of each row as an observation and each column as variables/measurments? –  user603 Nov 24 '12 at 18:52
show 1 more comment

1 Answer

How are you defining "outlier"? Looking at the example plot, I don't see any real outliers. There's just some noise in the data.

However, if you wanted to identify the points that were farthest from the fitted line, that would be fairly straightforward using the predict or residuals functions in the appropriate model. E.g.

x <- 1:100
y <- 3*x + rnorm(100)
m1 <- lm(y~x)
residm1 <- m1$residuals
ranks <- rank(residm1)

You could then select the largest n values for inspection or choose a minimum residual that would qualify as an "outlier".

share|improve this answer
    
It doesnt look like it will fit in a polynomial model? Or will it? How to find the best non-linear model which will fit here? –  Pankaj More Nov 24 '12 at 15:52
    
Search "curve fitting" for some ideas. –  Peter Flom Nov 24 '12 at 16:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.