Linear Model Equivalence regarding Latent Variables

Question

While reading a paper (surprises in high-dimensional ridgeless least squares interpolation), I was stuck to a part of a section (5.4, page 16~18).

Consider coviariates $x_i = (x_{i,1},\dots,x_{i,p}) \in \mathbb R^p$ and a response variable $y_i \in \mathbb R$ which are distributed as follows: $$y_i = \beta^Tx_i+\epsilon_i, \tag a$$ where $\mathbb E[x_i] = 0, \text{Cov}[x_i] = \Sigma$ and $\mathbb E[\epsilon_i] = 0, \text{Var}[\epsilon_i] = \sigma^2$. Also, we assume that the resopnse variable is linear in a latent features $z_i\in\mathbb R^d (p \ge d)$ and the covariates are also linear in $z_i$: $$y_i = \theta^Tz_i+\xi_i,\qquad x_{i,j}=w_j^Tz_i+u_{i,j}\rightarrow X=ZW^T+U,\tag b$$ where $\xi_i \sim N(0,\sigma_{\xi}^2), u_{i,j}\sim N(0,1)$ are mutually independent, independent of $z_i\sim N(0,I_d)$, and $\theta,w_j \in\mathbb R^d$. Letting $W\in\mathbb R^{p\times d}$ a matrix whose ith row is $w_i$, the models in (a) and (b) are equivalent. That is, we have the following results by matching their covariances: $$ \begin{align} \Sigma &=I_p+WW^T &(1)\\ \beta &= W(I_d+W^TW)^{-1}\theta & (2)\\ \sigma^2 &= \sigma_{\xi}^2+\theta^T(I_d+W^TW)^{-1}\theta &(3) \end{align}$$

The result (1) is quite straitforward by defining $x_i =Wz_i+u_i$ where $u_i\in\mathbb R^p, u_i\sim N_p(0,I)$ and $Cov[x_i] = WCov[z_i]W^T+I_p = WW^T+I$.

But I'm not sure whether the result (2) and (3) are also derived from matching the covariances. I tried to use the following method $$\begin{align} y_i &= \beta^Tx_i+\epsilon_i \\ &= \beta^T(Wz_i+u_i) + \epsilon_i \\ &= \theta^Tz_i+\xi_i, \end{align}$$ but I don't think this provides useful ideas regarding the desired results (2), (3).

Any hint with this questions would be grateful. Thank you.

Answer 1

It is verifiable, under the additional condition that $x_i$ and $\epsilon_i$ are uncorrelated.

To show $(2)$, let's compute $\operatorname{Cov}(x_i, y_i)$ in two ways -- one from each model specification. Specifically, by $y_i = \beta^\top x_i + \epsilon_i$, we have \begin{align*} & \operatorname{Cov}(x_i, y_i) = \operatorname{Cov}(x_i, \beta^\top x_i + \epsilon_i) = \Sigma\beta. \tag{A.1}\label{1} \end{align*}

On the other hand, by $y_i = \theta^\top z_i + \xi_i$ and $x_i = Wz_i + u_i$ (as well as independence between error terms), we have \begin{align*} & \operatorname{Cov}(x_i, y_i) = \operatorname{Cov}(Wz_i + u_i, \theta^\top z_i + \xi_i) = W\theta. \tag{A.2}\label{2} \end{align*}

Comparing $\eqref{1}$ and $\eqref{2}$ then yields $\beta = \Sigma^{-1}W\theta$. Inserting the relation $\Sigma = I_p + WW^\top$ which you have verified, it follows that \begin{align*} \beta = (I_p + WW^\top)^{-1}W\theta = W(I_d + W^\top W)^{-1}\theta. \end{align*} For a proof to the last equality sign, see the end of this answer.

Now let's show relation $(3)$ by computing $\operatorname{Var}(y_i)$ in two ways. By $y_i = \beta^\top x_i + \epsilon_i$ and the condition $\operatorname{Cov}(x_i, \epsilon_i) = 0$, we have \begin{align*} & \operatorname{Var}(y_i) = \operatorname{Var}(\beta^\top x_i) + \operatorname{Var}(\epsilon_i) = \beta^\top \Sigma\beta + \sigma^2. \tag{A.3}\label{3} \end{align*}

On the other hand, by $y_i = \theta^\top z_i + \xi_i$, we have \begin{align*} & \operatorname{Var}(y_i) = \operatorname{Var}(\theta^\top z_i) + \operatorname{Var}(\xi_i) = \theta^\top\theta + \sigma_\xi^2. \tag{A.4}\label{4} \end{align*}

Equating $\eqref{3}$ and $\eqref{4}$ then yields \begin{align*} \sigma^2 = \sigma_\xi^2 + \theta^\top\theta - \beta^\top\Sigma\beta. \tag{A.5}\label{5} \end{align*}

To eliminate $\beta$ from $\eqref{5}$, using $\Sigma\beta = W\theta$ and $\beta = W(I_d + W^\top W)^{-1}\theta$ to get \begin{align*} \beta^\top\Sigma\beta = \theta^\top(I_d + W^\top W)^{-1}W^\top W\theta = \theta^\top \theta - \theta^\top(I_d + W^\top W)^{-1}\theta. \tag{A.6}\label{6} \end{align*} Substituting $\eqref{6}$ into $\eqref{5}$ gives $(3)$.

---

Proof of the identity $(I_p + WW^\top)^{-1}W = W(I_d + W^\top W)^{-1}$.

By Woodburry matrix identity, \begin{align*} (I_p + WW^\top)^{-1} = I_p - W(I_d + W^\top W)^{-1}W^\top. \end{align*} Therefore, \begin{align*} &(I_p + WW^\top)^{-1}W = (I_p - W(I_d + W^\top W)^{-1}W^\top)W \\ =& W - W(I_d - (I_d + W^\top W)^{-1}) = W(I_d + W^\top W)^{-1}. \end{align*}