Is the PCA estimator used in regression root-n-consistent?

Question

Consider a sample of $n$ observations $(y_i, x_i)$, $i=1,\ldots,n$ and assume without loss of generality that the samples are centered. The true model is $y_i=x_i^t\beta+\epsilon$, and the OLS estimator, which is a $\sqrt{n}$-consistent estimator of $\beta$ is obtained as,

$$\hat{\beta}=(X^tX)^{-1}X^ty$$

Now perform a PCA on the matrix $X\in\mathbb{R}^{n\times p}$. Consider the matrix of principal components $Q\in\mathbb{R}^{p\times p}$ defined in a way such that the first principal component has the largest possible variance, and each succeeding component has the largest possible variance under the constraint that it is orthogonal to the preceding components. From an algebra perpective, $Q$ is the matrix of eigenvectors from $X$ and define an orthogonal change of basis maximizing the variance from $X$.

Consider $Z=XQ\in\mathbb{R}^{n\times p}$ the projection of $X$ into the subspace generated by $Q$ and solve,

$$\tilde{\beta}=(Z^tZ)^{-1}Z^ty$$

Where $y_i=z_i^t\beta+\epsilon$. We can now go back to the original subspace from $X$ (in order, for example, to ease the interpretation of the coefficients) $$\hat\beta=Q\tilde\beta$$

But what happens if we do not select all the principal components from $Q$ but just a subset of the first $d$, $d<p$ components and calculate $Z_d=XQ_d$?

Now if we solve $$\tilde{\beta_d}=(Z_d^tZ_d)^{-1}Z_d^ty$$ and then project back onto the original subspace $$\beta^*=Q_d\tilde\beta_d$$ Is $\beta^*$ a root-n consistent estimator of $\beta$? Is it even a consistent estimator? How does affect the number of components chosen to the consistency?

Answer 1

It will not be consistent if $Y$ is explained by any of the the discarded components.

Consider the case with $p=2$ and $ d = 1$ where the first principal component is $(1,0)$ and the second is $(0,1)$ where the first explains $\%99$ percent of the variancewith $(\beta_1, \beta_2) = (0, 100)$. There is no way $\beta_2$ can be estimated if all observation are mostly projected over the $x$-axis.

The following script simulates this scenario with $\sigma^2 = 0$ in the linear part of the model.

set.seed(123)
beta_true = c(0,100)

beta_est = matrix(0, ncol = 2, nrow = 1000)
sqe = rep(0, 1000)

for (k in 1:1000 ){ 
x = as.matrix(cbind(rnorm(1000, 0, sd = sqrt(0.99) ), rnorm(1000, 0, sd = sqrt(0.01))))
Y = x%*%matrix(beta_true , ncol =1)

q = princomp(x)  

b1 = lm(Y ~ q$scores[,1] -1)$coefficients # fit linear model with the projection in the first components
beta = b1*q$loadings[,1] # 

beta_est[k, ] =beta
sqe[k] = sum((beta_true - beta )^2)
}
boxplot(beta_est, main ='Estimation of beta')
msqe = mean(sqe) # 9999.896

This boxplot shows estimation of $\hat \beta_2 \approx 0$ which is way off for a sample size of $n = 1000$.