1
$\begingroup$

I am working with a dataset where I suspect there is significant noise in both the dependent variable ($y_t^*$) and the independent variable ($x_t^*$). I am considering using moving averages to denoise the data before performing ordinary least squares (OLS) regression. I would appreciate insights into whether this approach is statistically sound and the implications it might have on the analysis.

The true underlying model of my data is:

$$ y_t = kx_t + b + e_t $$

However, the observed variables are:

$$ y_t^* = y_t + v_t $$ $$ x_t^* = x_t + \mu_t $$

where $v$ and $\mu$ represent measurement errors, and $e$ is the error term in the regression model. All these errors ($v$, $\mu$, and $e$) are assumed to be independent and identically distributed (iid).

In a standard OLS regression with $y^*$ and $x^*$, the covariance term $\text{cov}(y^*, x^*)$ remains as $k \times \text{var}(x)$, but the variance $\text{var}(x^*)$ becomes $\text{var}(x) + \text{var}(\mu)$, which would lead to a shrinkage in the estimated coefficient for $x$.

I am considering using a rolling average to mitigate this issue. Assuming that $x_t$ are stationary but autocorrelated, and $\mu_t$ is iid with no autocorrelation, the formula for the average would give:

$$ \frac{\text{cov}(\bar{y}^*, \bar{x}^*)}{\text{var}(\bar{x}^*)} = \frac{k(\text{var}(x) + \text{[some linear combination of autocorrelation terms]})}{(\text{var}(x) + \text{[some linear combination of autocorrelation terms]}) + \text{var}(\mu)} $$

If $x$ is highly autocorrelated and we take a larger rolling window, this increases the long-run variance, potentially outweighing $\text{var}(\mu)$, which remains constant. Would this approach yield an estimate closer to the true coefficient $k$?

Furthermore, I am curious about the implications of resampling the data at a lower frequency (e.g., converting daily data to weekly or monthly) on the variance. Also, what would be the effect if the rolling average is applied only to $x^*$ and not to $y^*$, especially if the measurement error is primarily in $x^*$?

I am seeking insights on whether my analysis is correct and if the proposed methodologies are valid approaches to handle noisy data in regression analysis.

$\endgroup$
8
  • $\begingroup$ Are you trying to ask how to do a form of errors-in-variables regression? In the middle your question seems to change: all of a sudden it switches from regression to some form of time series analysis. It's unclear what's going on and what you really aim to do. $\endgroup$
    – whuber
    Feb 1 at 23:18
  • $\begingroup$ Sorry I should specify that my data are time-series. $\endgroup$
    – The One
    Feb 1 at 23:42
  • $\begingroup$ I am tempted to suggest setting up a state-space model, as you setup looks very much like one. It should not be too much work and it would yield point estimates as well as estimates of uncertainty. $\endgroup$ Feb 5 at 11:58
  • $\begingroup$ @SextusEmpiricus, regarding consistency, isn't there bias as per OP's equation for $\frac{\text{cov}(\bar{y}^*, \bar{x}^*)}{\text{var}(\bar{x}^*)}$? $\endgroup$ Feb 12 at 7:32
  • $\begingroup$ @SextusEmpiricus, OK, so it is consistent unless there is measurement error in $x$ which the OP is actually concerned with. $\endgroup$ Feb 12 at 7:44

2 Answers 2

1
$\begingroup$

I have worked with a device (a pressure transducer) with a built-in feature to take multiple measurements very fast and then average them to reduce measurement errors exactly for the reasons you stated.

One difference you need to pay attention to is that the measurements are taken much faster than the process physical time. In that case (where autocorrelation ~ 1) it's a great idea.

You seem to imply, as it wasn't stated directly, that the measurements are equispaced and in a time scale relevant to your physical process. In this case you should consider what you're doing carefully as averaging effectively changes the time of the sample:

$$ (y_{t-1} + y_{t-2})/2 \approx y_{t-1.5} $$

and for 3:

$$ (y_{t-1} + y_{t-2}+ y_{t-3})/3 \approx y_{t-2} $$

so the longer the average the farther back in time your apparent measurement will be.

Another way you should consider introducing past samples is with autoregression. Instead of the simple model you stated:

$$ y_t=kx_t+b+e_t $$

you want to use a mean for $x_t$:

$$ y_t=k(x_t+x_{t-1})/2+b+e_t.$$

Well then the question is, why force the weight of $x_t$ and $x_{t-1}$ to be equal? Why not just use

$$ y_t=k_1x_t+k_2x_{t-1}+b+e_t?$$

This naturally allows you to treat the effect of the time lag between samples. In case you just want to find the coefficients $k$ and $b$ then no averaging is needed as it's already accounted for in the OLS and your uncertainty in the coefficients.

$\endgroup$
1
  • $\begingroup$ Keep in mind that $x$ is unobservable. We measure it with error and observe $x^*$ instead. $\endgroup$ Feb 11 at 19:19
1
$\begingroup$

The variance of a sum of $n$ variables, is the sum of the terms in their covariance matrix. If this sum of the terms increases relatively faster than $n$, which is the variance of the sum of the error terms, then the bias will be reduced.

For an AR1 process with marginal variance 1, this covariance matrix is a symmetric Toeplitz matrix with terms $\rho^0,\rho^1,\rho^2, \dots , \rho^{n-1}$. And for the variance of a sum of $n$ AR1 variables we have:

$$\text{var}\left(\sum{x}\right) = \left(n+ 2 \sum_{k=1}^{n-1} \frac{n-k}{n} \rho^k \right) = 2 \frac{\rho(\rho^n-1)}{(1-\rho)^2} + n \left(1+ \frac{2\rho}{1-\rho}\right) $$

If $x$ is highly autocorrelated and we take a larger rolling window, this increases the long-run variance, potentially outweighing $\text{var}(\mu)$, which remains constant. Would this approach yield an estimate closer to the true coefficient $k$?

The use of an average can decrease the bias somewhat, but because the behaviour towards infinity is still $O(n)$, we can not decrease the bias indefinitely. The estimator is not consistent (unlike what I mentioned before in the comments)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.