Assuming we have $n$ principal components and use $k<n$ for a linear regression. What is the bias of the l.s.e estimator $\hat \beta$ for the slope parameter using just these k components of the transformed design matrix instead of the full transformed design matrix ?


I am talking about a model $y = Z\beta+\epsilon $

Where $Z$ is the transformation of the design matrix $X$ to the space of $k$ principal components. If I would take $k=n$ than $\hat \beta$ would be equivalent to the least square estimate of the original data. But since I cut off some principal components I lower the variance but introduce a bias. Therefore I want to calculate the bias with respect to the l.s.e of the untransformed data with the model: $y = X \gamma+\epsilon$

  • 1
    $\begingroup$ Could you define the estimand $\beta$ and, better yet, the model? Otherwise it is not clear with respect to what the bias is to be derived. $\endgroup$ Nov 26, 2018 at 20:42
  • $\begingroup$ Perhaps PLS regression would be of more interest to you. $\endgroup$
    – Carl
    Nov 26, 2018 at 21:03
  • $\begingroup$ Since each of the principal components are orthogonal, there is no bias from using the first $k$ columns as regressors rather than all columns of the design matrix representation. The formula for omitted variable bias shows that the bias depends on the correlation between the two predictors. On the other hand, you have to rely on the intrinsic meaning of the principal components for interpreting the $\beta$s and the predictions will be inefficient (high MSE). $\endgroup$
    – AdamO
    Nov 28, 2018 at 15:52
  • $\begingroup$ @AdamO I dont believe that " there is no bias from using the first k columns as regressors rather than all columns of the design matrix representation." Imagine using just one principal component of a high dimensional dataset for regression. You will definetly have a bias, when the full model $ y = X\gamma + \epsilon$ is unbiased. $\endgroup$
    – PascalIv
    Nov 29, 2018 at 16:06
  • $\begingroup$ @PascalIv then you know the answer to your question? Maybe a little proof or simulation to show what you mean? Give a thought to what bias means here. For the coefficients to the PCA, it doesn't matter. All k+ eigenvectors are orthogonal to the first k. $\endgroup$
    – AdamO
    Nov 29, 2018 at 16:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.