0
$\begingroup$

I've been reading a bunch of posts that advise people to not include test data when preprocessing. So I've proceeded by first setting aside a test dataset to be used to assess how well my classifier at the final stage.

I have two sources of data: clinical data and gene expression data. Since the gene expression data has many, many dimensions, I wanted to use PCA to reduce the number of dimensions to about 15. Then I combined the PCs with the clinical predictors to obtain a "full" features dataset. Then, I trained a model using train in the caret package. However, I want to use that classifier to make predictions, but when I put my test dataset into the predict function, I get an error because the principal components are not in the test dataset.

So my question is, do I run PCA on the test dataset as well? Or am I misunderstanding the whole concept of preprocessing? Thank you!

$\endgroup$
1
$\begingroup$

You have found the principal component direction, say $v_1, \ldots, v_k$, from the training data set. Let $V= [v_1, \ldots, v_k] \in \mathbb{R}^{n \times k}$

Given any data point $x \in \mathbb{R}^n$ (assuming they have been centralized), we can project it to a $k$-dimensional space by computing $V^Tx \in \mathbb{R}^k$.

While $V$ is obtained only from the training data, give any data, including training, validation, or test data, we can perform such projection, and hence we can compute $V^Tx_{\text{test}}$ and make it compatible with the training data.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry if I'm misinterpreting your comment. Are you suggesting that I run a separate PCA for the test dataset? $\endgroup$ – user122514 Feb 28 '18 at 0:18
  • $\begingroup$ you have obtained the principal component direction/ also known as eigenvectors from the training data right, we can use those direction to compute its projection for the test data as well. $\endgroup$ – Siong Thye Goh Feb 28 '18 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.