Can I take a random sample of my very large data set to overcome non-independence?

Question

I am trying to run a regression model on a very large time series data set (comparing flow noise to vehicle speed, pitch and dive state). Because my samples are taken about every minute (with some gaps of a few hours, so not perfectly spaced), the independence assumption is violated.

I have selected a GLS model because I am also dealing with non-constant variance. I built in a variance structure to deal with that. I tried to build in a covariance structure, but the run times are not practical.

I have 15,000+ data points, so my thought was I could take a random sample of my 15,000 points and now the independence assumption is satisfied, and I can run the GLS just with the variance structure. I could run 1000+ iterations with new random samples each time and compare.

My question is - is this valid? Am I missing anything here?

And as a follow up - my data also has outliers. I am removing them using Cook's distance. Should that be done before or after taking the random sample?

Answer 1

If you downsample time-series data it would not remove the dependence, it would just dilute it. Say that your data follows the relationship

$$ y_{t+1} = f(y_{t}) + \varepsilon_{t+1} $$

so the current time-point directly depends on the previous time-point. What is the relationship between $y_t$ and $y_{t+k}$? Well, it's

$$ y_{t+k} = f( \dots f(f(f(y_{t}) + \varepsilon_{t+1}) + \varepsilon_{t+2}) + \varepsilon_{t+3}) \dots) + \varepsilon_{t+k} $$

so there is still some dependence on the value $k$ steps before, but the relationship is much noisier than with the value one step before. So if you remove the intermediate steps, there would be still some dependence. For exactly the same reason, thinning the MCMC chains, which once was a standard approach, is not recommended anymore as it does not "remove" the dependency.

That said, regression models are pretty common also when dealing with time-series data. In such a case, you are usually looking at the data "one row at a time" and the model is

$$ y_t = \mathbf{X}_t\boldsymbol{\beta} + \varepsilon_t $$

so it doesn't consider the past or the future. The assumption is that the errors are independent. If they are not, you would need a model that allows for auto-correlated errors, i.e. a time-series model like dynamic regression. There are ways to deal with it without a time-series model, but time-series models are the tools for the job.

Answer 2

I will add a couple of points to Tim's answer, focussing on the original question, which was "My question is - is this valid? Am I missing anything here?". I think the approach can be valid, but... it depends on lots of details.

Regarding the question of whether you "remove" or "dilute" the dependence when you downsample I feel this is more about semantics than anything else. If you downsample enough you will get to a point where for all practical purposes you have removed the correlation. So if you had lots and lots of data and enough signal on them, that could certainly be a pragmatic approach. And if the residuals of the model showed no "significant" correlation after the fitting, then you'd be home safe. In a way... because, nonetheless, that is sub-optimal for at least 4 reasons:

1. By throwing away data you are throwing away useful information
2. This introduces an arbitrary treshold via the chosen downsampling factor (which if you downsample too much, might actually limit your ability to find existing significant effects, say
3. Depending on the sample you select out of the complete data set you could end up with different results, so by doing it you are really ignoring a component of the variability in the reported estimates (but then again, this could be negligible, and you could try to work around it by doing this many times via a resampling procedure)
4. You might be interested in the correlation structure itself (although I assume this is not the case for you)

So I would not dismiss it if it saves you the headaches I suspect might come along with properly fitting a model with the correct correlation structure at the scale you collected the data. On the other hand, you should think that if you had collected the data once every 10 minutes you would only have 1/10th of the data any way. So I guess an important factor to consider is at what sclae are you interested in describing the phenomena you are trying to model? Nowdays, with some sensors available to researchers, say, sound sensors sampling at 256kHz, one might argue that for practical purposes throwing away large amounts of data might be OK. This will always be an "it depends on the specific setting" question, so I'm not really advocating one might do that as a blind routine. But it might be a pragmatic effecient approach.

Not really relevant for the point being discussed, but commenting on a point that @tim noted, I believe the reason that thinning is not recommended in an MCMC analysis is not that there is still correlation after you thin, but because this is always an inneficient procedure since you use the vales of the MC as a posterior distribution, and essentially you actualy marginalize over time (you ignore the temporal autocorrelation). Hence, removing values, even if they are correlated, is always less eficient that using all the values (this provided the chain has converged, but if it has not, no need to try to make inferenc3es out of it). Then again, I have like 61 reputation points and @tim does have thousands, so he might know stuff I don't ;) !