Consider a linear model $y_n = \beta x_n + v_n$ where $n$ represents the time index, $x$ is the input data source $\hat{\beta}_{OLS} = (X^TX)^{-1}X^Ty$ is the ordinary least square estimate of the parameter $\beta$. This parameter is obtained assuming the distribution of the observation $y$ is $i.i.d$ Gaussian.

Consider, the covariance of the estimate $\hat{\beta} = \sigma^2{(X^TX)}^{-1}$

The answers to the following questions are extremely hard for me to obtain from books, therefore I have posted the question. Shall appreciate help. Also, please correct me if any information is wrong.

1) What is the meaning of autocorrelation and how it will change in the following scenario -- If the measurement noise $v$ is White Gaussian then it means that the noise samples are $i.i.d$. If the measurement noise is colored Gaussian, then the noise samples are not $i.id$ but correlated which means that the property of independence and uncorrelated samples cannot be applied. When the noise is white Gaussian then expectation, $E[vv^T] = \sigma^2$ I think. What does the autocorrelation function look like in white and colored Gaussian case and what would expectation of colored noise samples give?

2) UPDATE: Consider, a new data source denoted by $x_{new}$ which has some different properties than $x$.

If the covariance $\sigma^2{(X^TX)}^{-1}$ is higher than another estimator of $\hat{\beta}$ given as $\hat{\beta_{new}} = \sigma^2{(X_{new}^TX_{new})}^{-1}$ obtained using the new data input $x_{new}$, then what does having a lower variance say about the nature of the data $x_{new}$? Can I say that $x_{new}$ input transfers more energy, has higher power than the data $x$?

3) Does $X^TX$ term have a name?

  • 1
    $\begingroup$ Saying that $y$ is i.i.d and then wondering about autocorrelation, doesn't make sense - they are mutually exclusive. I am further confused because the model you propose is not indexed over time, so where would the autocorrelation come from? Finally, the class of linear of estimators is very very broad, and again, for the proposed model there is no point in switching to a non linear estimator. Maybe you can give us some to context? What are you trying to do? What lead you to wonder about these questions? $\endgroup$ – Repmat Dec 16 '17 at 11:32
  • $\begingroup$ @Repmat: I have updated the question, thank you for your suggestions. $\endgroup$ – Ria George Dec 16 '17 at 17:23

Disclaimer: I'm not entirely clear on everything myself, this is my best-effort, open for improvements.

  1. If $$y_{n + 1} = \beta\cdot y_n + \epsilon_n$$ and $$\epsilon_n \sim \text{norm}(0, 1)$$, then autocorrelation can be simplified by linearity and because the errors are independent, $$\text{cov}(y_n, y_{n - 1}) = \beta\cdot\text{cov}(y_{n-1}, y_{n-1}) = \beta\cdot\text{var}(y_{n-1})$$. If $\beta$ is zero then this is "white noise" and the autocorrelation is zero. For larger gaps the same simplifications can be used to show $\text{cov}(y_n, y_{n-k}) = \beta^k\cdot\text{var}(y_k)$.

  2. OLS is the maximum likelihood estimate of $\beta$ if the error is Gaussian. But if noise is non-Gaussian, then this may not work that well. Let's say the noise is skewed "upwards", so errors are a bit bigger upwards, let's say lognormal, then the maximum likelihood estimate for the mean is different. So if you use it, it's not optimal.

  3. If you are estimating the mean of a lognormal distribution, see the link above for the nonlinear estimator of that.

EDIT, some topics below

  • I think the formula in the original question changed, now it's $y_n = \beta\cdot x_n + \epsilon_n$. Unless there is information on how $x_n$ or $y_n$ depend on variables at time $n-1$, there is not much to say about autocorrelation.
  • Autocorrelation is correlation with previous values, which is covariance with previous values divided by standard deviations. To simplify, I've only shown a derivation with covariance.
  • In the comments below, I think you mean variance of $\beta$, not it's covariance (otherwise, covariance with what?).
  • The assumptions of the question regarding $X_{\text{new}}$ are unclear to me, and I suspect it's a question I don't necessarily know the answer to, or maybe there is not a single answer. Basically, if you assume Gaussian error, then the OLS estimate is the maximum likelihood estimate and also the unbiased estimator of $\beta$ with the lowest possible variance. If you change the distribution of the noise, then the bets are off. Maybe you can start with assuming different noise model and the see what happens, before trying to get the general answer. To be precise, the OLS estimate is the best unbiased estimator under broader assumptions, see https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.
  • $\begingroup$ Thank you for your answer. You are correct by indexing the variables with respect to time. My autocorrelation for the noise in the model in the question is $v_n v_n^T$. I don't follow why you have covariance when explaining autocorrelaton. For Point 2) assume I am estimating a parameter using the same dynamical model by assuming different distribution of the noise. I want to select the best estimate of the parameter. $\endgroup$ – Ria George Dec 16 '17 at 15:27
  • $\begingroup$ The variance of the estimate should be minimum in order to make this choice. Can variance of the estimate be less when the distribution is non-Gaussian and what happens to autocorrelation for non-Gaussian case? $\endgroup$ – Ria George Dec 16 '17 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.