# How can I remove autocorrelation from this data?

I have 1000 data points and can see there is quite a high level of autocorrelation.

If I look at the lag 1 data pairs (1, 2), (3, 4), (5, 6), ..., then the correlation is r = .62.

I put the raw data here. I'd like to transform the data to remove or reduce the autocorrelation.

The context for all this is that the data points are guesses made by individuals about some quantity. I know the true value of the quantity and want to see whether the average guess is better if I just leave the data autocorrelated, or if I remove the autocorrelation.

One idea I had for an approach was to get rid of the lag 1 autocorrelation first, and then to see if average accuracy improves, and then try to get rid of the lag 2 autocorrelation.

I thought maybe I could just throw out every second data point, and then naturally autocorrelation would reduce. However, ideally I'd like to find a way to remove autocorrelation without throwing away data.

• Is that time series data? Or is that 1000 people guessing once each? – Richard Hardy Nov 28 '15 at 21:54
• It's 1000 people guessing once each, one after the other. Each person gets to see only the person who guessed directly before them. So, if you're Person 100 you got to see what Person 99 guessed, before making your own guess. You might be influenced by Person 99's guess, or you might decide to ignore it. – user1205901 - Reinstate Monica Nov 28 '15 at 22:58

The setting looks like a regression with ARMA errors, more precisely AR(1) errors (by construction, as I understand the data generating process from your comment). It can be estimated by function arima in R specifying ARIMA order (1,0,0) and including any regressors as exogenous regressors via the argument xreg. The adjusted data could be obtained as residuals from the model using residuals(model) where model is the estimated model, plus the intercept of the model (since in arima intercept is actually the mean, see here, ISSUE 1).

Your approach of deleting data point is not a very good idea. You will lose a lot of information but the autocorrelation will still be there.

• I have my vector of 1000 values, called A. Then I run arima(A,order=c(1,0,0)), and get the coefficients ar1=0.63 and intercept=0.08. How do I interpret those coefficients to create an adjusted version the data with the autocorrelation gone? Ultimately I want to see if I can get a better estimate (either closer to true value, or lower-variance) of the data generating process from some adjusted version of the data than from the actual data. – user1205901 - Reinstate Monica Nov 30 '15 at 2:38
• I edited the post to show how to get adjusted data. Meanwhile, coefficient interpretation is a different question; you may find similar questions elsewhere on Cross Validated, hopefully they will fit your case. – Richard Hardy Nov 30 '15 at 7:23
• As an example, I generate autocorrelated data with x <- filter(rnorm(1000), filter=rep(1,3), circular=TRUE)+2. So the mean of the data generating process is 2, but due to the autocorrelation the mean of x tends to be out by .1 or so. Then I run model <- arima(x,order=c(1,0,0)) and I can see the residual value for each data point. How do I get from those residuals to the adjusted data? – user1205901 - Reinstate Monica Nov 30 '15 at 11:43
• The residuals are the adjusted data, except for the intercept (my bad, I will correct the answer). The fitted values (minus the intercept) are the effects of autocorrelation. – Richard Hardy Nov 30 '15 at 12:04
• I wrote some R code (at pastebin.com/0w1AnZep) to test all of this, and I found that the mean error of just taking the average of the raw data was .241, and the mean error of taking the average of the transformed data was .244. However, the SD is much lower in the transformed data. Is it to be expected that transforming the data would produce no improvement in mean error, but a reduction in variance? Or is it a sign I've implemented your idea incorrectly? – user1205901 - Reinstate Monica Nov 30 '15 at 14:39
dta <- read.table("anchoring.csv", col.names="v1")
summary(lm1 <- lm(dta$v1[-1]~dta$v1[-1000]))


This is an AR(1) regression, estimated using lm instead of arima. The summary shows that ~40% of the variance in the persons' guesses can be explained by the guesses they were shown (that is the R squared).

predict(lm1)


will give you the expected value of a subject's guess, given the guess shown to them.