# Linear model with two columns where one column is a linear transformation of another

If we have a linear model

$$y_t = β_1x_{1t} + β_2x_{2t} + e_t$$

where the errors $$e_t$$ are i.i.d normally distributed: $$e_t$$ ~ $$IIN(0, σ^2)$$, t = 1,...,T. The vectors of random regressors $$X_1$$ and $$X_2$$ satisfy the relation $$X_1 = c + aX_2$$ where a and c are constants. Both regressors are uncorrelated with error vector e. All variables in the model (i.e. the dependent variable and the regressors) have mean of zero.

(a) Find the covariance and correlation between $$X_1$$ and $$X_2$$

(b) Write the matrix X'X and show that it is non-invertible.

I think the covariance is

$$Cov(X_1, X_2) = Cov(c + aX_2, X_2) = aVar(X_2)$$

and I think the correlation is

$$Corr(c + aX_2, X_2) = \frac{aVar(X_2)}{a^2Var(X_2)} = a$$.

Is that correct? As for showing X'X is not invertible, I'm lost. Everything I've tried shows that X'X is, in general, invertible but the premise of the question seems to say I am wrong about that.

• Whoever posed this question might have forgotten to include an intercept in the model.
– whuber
Commented Oct 20, 2018 at 22:53
• It's a question given me in an assignment that I don't understand. I presented the question as given. Commented Oct 20, 2018 at 23:15

$$\mathrm{Var}(X_1)=\mathrm{Var}(c+aX_2) = a^2\mathrm{Var}(X_2)$$

$$\mathrm{Corr}(X_1,X_2)= \frac{\mathrm{Cov}(X_1,X_2)}{\sqrt{\mathrm{Var}(X_1)\mathrm{Var}(X_2)}} \\= \frac{a\mathrm{Var}(X_2)}{\sqrt{a^2\mathrm{Var}(X_2)\mathrm{Var}(X_2)}} = \text{sign}(a)$$

$$X=(X_1, X_2) =(c+aX_2, X_2)$$

$$X'X = \left(\matrix{(nc^2 + 2ac\sum X_2 +a^2\sum X_2^2 & c\sum X_2+a\sum X_2^2\\ c\sum X_2+a\sum X_2^2& \sum X_2^2}\right)$$

Let $$X_2 = (1,2,3)'$$ and $$X_1=1+X_2=(2,3,4)'$$. Then $$X'X=\left(\matrix{29& 20 \\20 & 14} \right)$$

$$(X'X)^{-1} =\frac 16\left(\matrix{14& -20\\ -20& 29}\right)$$

So that $$(X'X)$$ is non-invertible is not true.

• Thanks! That is exactly what I thought (aside from messing up the calculation of correlation). Is it that the question is based on an incorrect premise? Commented Oct 20, 2018 at 23:18
• if model is $y_t = \beta_0 + β_1x_{1t} + β_2x_{2t} + e_t$, and $X$ is design matrix including intercept term ($x_{i1} = 1$ for all $i$). Then $X$ should be $n\times 3$ matrix. and this $n\times 3$ $X$ is non-invertible. My latex improved a lot by this answer. Commented Oct 20, 2018 at 23:23

Covariance is indeed equal to

$$Cov(c + aX, X) = aVar(X)$$

As for the correlation, assuming $$a \neq 0, Var(X) \neq 0$$, there is an important detail

$$Cor(c + aX, X) = \frac{Cov(c + aX, X)}{\sqrt{Var(c + aX)}\sqrt{Var(X)}} = \frac{aVar(X)}{|a|Var(X)} = sgn(a)$$

As for the non-invertibility, model without the bias term, i.e.

$$Y = \beta_1X_1 + \beta_2X_2 + \epsilon$$

indeed has an invertible design matrix $$X$$, if $$c \neq 0$$.

However, if you add the bias

$$Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \epsilon$$

then the design matrix $$X \in R^{n \times 3}, n \geq 3$$ is no longer invertible, since it does not have rank $$3$$.

• You appear to have mixed up the roles of $a$ and $c$ initially. This error is then propagated into subsequent formulas. The remainder of your reply seems to assume $X$ is fixed rather than random, but the question stipulates they are random.
– whuber
Commented Oct 20, 2018 at 22:48
• Yes, mixing up happens when I copy paste the formulas... As for the notation, it may be sloppy, but I do not assume $X$ is fixed. $X_1$ and $X_2$ denote random variables, not columns of the design matrix $X = (\vec{x_1}, \vec{x_2})$. E.g. $\vec{x_1}$ are realizations of $X_1$ Commented Oct 21, 2018 at 7:56