Why is autocorrelation so important? I've understood the principle of it (I guess..) but as there are also examples where no autocorrelation occurs I wonder: Isn't everything in nature somehow autocorrelated? The last aspect is more aiming at a general understanding of the autocorrelation itself because, as I mentioned, isn't every state in the universe dependent on the previous one?
1$\begingroup$ I like this question, although it's a bit too philosophical from my point of view :) I can give some historical context, which might be of help. I think as far as signal processing is concerned it has a lot to do with spectral estimation. Look into spectral estimation and power spectral densities from finite amounts of data. This might give you an idea of why autocorrelation is (or rather was) so important. $\endgroup$– idnavidSep 16, 2019 at 6:22
4$\begingroup$ I don't understand the question in your title. There's no purpose, it's just a property of data that has to be accounted for in some types of analyses. Why it is important is probably answerable. $\endgroup$– mktSep 16, 2019 at 6:31
2$\begingroup$ Isn't everything in nature somehow autocorrelated? Phenomena that are not time series would not be autocorrelated, because autocorrelation is a property of a time series (though there are notions of spatial correlation and other to reflect relationships along dimensions other than time). But since everything is taking place in time, autocorrelation might indeed be pretty ubiquitous. $\endgroup$– Richard HardySep 16, 2019 at 9:52
3$\begingroup$ If everything in nature is somehow autocorrelated, then it sounds to me like autocorrelation is quite a big deal! $\endgroup$– DavidSep 16, 2019 at 11:18
2$\begingroup$ "Why is autocorrelation so important?": Prospecting time crystals, of course! $\endgroup$– NatSep 17, 2019 at 10:52
Autocorrelation has several plain-language interpretations that signify in ways that non-autocorrelated processes and models do not:
An autocorrelated variable has memory of its previous values. Such variables have behavior that depends on what went before. Memory may be long or short relative to the period of observation; memory may be infinite; memory may be negative (i.e. may oscillate). If your guiding theories say the past (of a variable) remains with us, then autocorrelation is an expression of that. (See, for example Boef, S. D. (2001). Modeling equilibrium relationships: Error correction models with strongly autoregressive data. Political Analysis, 9(1), 78–94, and also de Boef, S., & Keele, L. (2008). Taking Time Seriously. American Journal of Political Science, 52(1), 184–200.)
An autocorrelated variable implies a dynamic system. The questions we ask and answer about the behavior of dynamic systems are different than those we ask about non-dynamic systems. For example, when causal effects enter a system, and how long effects from a perturbation at one point in time remain relevant are answered in the language of autocorrelated models. (See, for example, Levins, R. (1998). Dialectics and Systems Theory. Science & Society, 62(3), 375–399, but also the Pesaran citation below.)
An autocorrelated variable implies a need for time series modeling (if not dynamic systems modeling also). Time series methodologies are predicated on autoregressive behaviors (and moving average, which is a modeling assumption about the time-dependent structure of errors) attempting to capture salient details of the data generating process, and stand in marked contrast to, for example, so-called "longitudinal models" which simply incorporate some measure of time as a variable in an otherwise non-dynamic model without autocorrelation. See, for example, Pesaran, M. H. (2015) Time Series and Panel Data in Econometrics, New York, NY: Oxford University Press.
Caveat: I am using "autoregression" and "autoregressive" to imply any memory structure to a variable in general, regardless of short-term, long term, unit-root, explosive, etc. properties of that process.
An attempt at an answer.
Autocorrelation is no different than any other relationship between predictors. It's just that the predictor and the dependent variable happen to be the same time series, just lagged.
isn't every state in the universe dependent on the previous one?
Yes indeed. Just as every object's state in the universe depends on every other object's, via all kinds of physical forces. The question just is whether the relationship is strong enough to be detectable, or strong enough to help us in predicting states.
And the very same thing applies to autocorrelation. It's always there. The question is whether we need to model it, or whether modeling it just introduces additional uncertainty (the bias-variance trade-off), making us worse off than not modeling it.
An example from my personal work: I forecast supermarket sales. My household's consumption of milk is fairly regular. If I haven't bought any milk in three or four days, chances are high I'll come in today or tomorrow to buy milk. If the supermarket wants to forecast my household's demand for milk, they should by all means take this autocorrelation into account.
However, I am not the only customer in my supermarket. There are maybe another 2,000 households that buy their groceries there. Each one's milk consumption is again autocorrelated. But since everyone's rate of consumption is different, the autocorrelation at the aggregate is so much attenuated that it may not make sense to model it any more. It has disappeared into the general daily demand, i.e., the intercept. And since the supermarket doesn't care who it sells milk to, it will model aggregate demand, and probably not include autocorrelation.
(Yes, there is intra-weekly seasonality. Which is a kind of autocorrelation, but it really depends on the day of the week, not on the demand on the same weekday one week earlier, so it's more a weekday effect than seasonal autocorrelation.)
2$\begingroup$ +1. Very nice example of how autocorrelation might be diminished in the aggregate. Just as a mixture of distributions can blur together and confound things. (And I've always thought that retail sales forecasting would be a cool job!) $\endgroup$– WayneSep 16, 2019 at 15:59
29$\begingroup$ @Wayne: it is. I tell my kids that daddy makes sure there is always enough ice cream at the supermarket. I think they love me a little more because of my job. $\endgroup$ Sep 16, 2019 at 18:54
First, I think you mean what is the purpose of evaluating autocorrelation and dealing with it. If you really mean the "purpose of autocorrelation" then that's philosophy, not statistics.
Second, states of the universe are correlated with previous states but not every statistical problem deals with previous states of nature. Lots of studies are cross-sectional.
Third, do we need to model it when it is there? Methods make assumptions. Most forms of regression assume no auto-correlation (that is, the errors are independent). If we violate this assumption, then our results could be wrong. How far wrong? One way to tell would be to do the usual regression and also some model that accounts for autocorrelation (e.g. multilevel models or time series methods) and see how different the results are. But, I think generally, accounting for auto-correlation will reduce noise and make the model more accurate.
3$\begingroup$ "then that's philosophy, not statistics." Eh... are you sure you want to make so sharp a distinction? After all, both statistical methodologists and philosopher's of science care about, for example, the distinctions between "prediction" and "explanation," in ways that are germane to the whys and wherefores of autocorrelated models. $\endgroup$– AlexisSep 16, 2019 at 23:44