1
$\begingroup$

Can a degenerate random variable (RV) have a non-zero covariance with a non-degenerate RV (or even degenerate, too)?

My intuition says "no" because that would imply (would it?) that values sampled from the degenerate variable and the non-degenerate have some mutual information, i.e. one can draw conclusions about samples from non-degenerate RV from samples from degenerate RV.

Now, I have to admit that there is a chance that a question is actually about me making mistakes in my algebra, however, I came across a case where it appears so:

I examine two linear models, namely: $$(1)\ y = X \beta + \epsilon_1+ g \ \ and \ \ (2) \ y = \epsilon_2 + g\\ \epsilon_1 \sim \mathbb{N}(0, \sigma_1^2 I_n), \epsilon_2 \sim \mathbb{N}(0, \sigma_2^2 I_n),\\ g \sim \mathbb{N}(0, \Omega), $$ where $\Omega$ is some known covariance matrix. Furthermore, $X$ is a square $n \times n$ matrix, i.e. $(1)$ is just a system of linear equations and therefore $\sigma_1 = 0$, so $\epsilon_1$ is degenerate. Now, I want to study covariance between $\epsilon_1$ and $\epsilon_2$: $$cov(\epsilon_1, \epsilon_2) = E[(\epsilon_1 - E[\epsilon_1])(\epsilon_2 - E[\epsilon_2])^T] = E[\epsilon_1 \epsilon_2^T] = E[(y- X \beta - g)(y- g)^T] = \\= E[y y^T - X \beta y^T +g g^T] =y y^T - X \beta y^T + \Omega$$ I could argue that as $X \beta$ is numerically $y$, then $y y^T -X \beta y^T = 0$, but there is still $\Omega$.

$\endgroup$

1 Answer 1

1
$\begingroup$
  1. In relation to your $y$ I suggest you not use the exact same symbol for distinct quantities; a subscript would do

  2. However, we can readily answer the title question; if a variable doesn't vary, it can't co-vary (vary with something else). The covariance is definitely 0, the correlation is 0/0 (undefined). The covariance of $\epsilon_1$ and $\epsilon_2$ doesn't involve $\Omega$; fixing your notation problem in 1 will probably help you identify that issue.

$\endgroup$
4
  • $\begingroup$ Could you please elaborate the notation part? I don't think I can use distinct symbols for $y$ in (1) and (2) as that would imply that I am trying to explain different quantities with 2 linear models, but I try to explain the same quantity ($y$) with 2 different linear models. $\endgroup$
    – Geo
    Commented Aug 5, 2020 at 1:29
  • $\begingroup$ In that case - assuming you intend both the error terms and the $g$ variable to be independent, since that's how you seem to be calculating with them - at most one of the models can actually be correct; if one is correct the other is untrue and then your calculations with them as random variables don't make sense. [You can fit an incorrect model, of course, but that's a different situation] $\endgroup$
    – Glen_b
    Commented Aug 5, 2020 at 1:41
  • $\begingroup$ Correct /true in what sense? If one model diverges from reality more, I would expect it just having greater σ. The thing is that I am interested in seeing how much common variance there is between their error vectors, i.e. variance that is left unexplained by both model (1) and model (2). $\endgroup$
    – Geo
    Commented Aug 5, 2020 at 1:51
  • $\begingroup$ You keep conflating something fitted to data with a question that is purely about random variables (not estimates). You have to stop doing that or the conversation cannot get anywhere. The relationships between random variables that you define along with the implied independence of $g$ from the error terms cannot all be simultaneously as stated; the mathematical relationships between variables have to all hold if you want to calculate with them as you did. You are defining two y-generating processes but they cannot both generate the same distribution for random variable y $\endgroup$
    – Glen_b
    Commented Aug 5, 2020 at 1:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.