Multivariate analysis and PCA

Question

I observe y as a function of x. y is a scalar, x is a vector of dimension n. I have k observations of y and x.

Let $$X = \begin{bmatrix}x_{1,1} & \cdots & x_{1,n}\\\vdots & \vdots & \vdots \\ x_{k,1} & \cdots & x_{k,n}\end{bmatrix}$$

$$Y = \begin{bmatrix}y_1 & x_{1,1} & \cdots & x_{1,n}\\\vdots &\vdots & \vdots & \vdots \\ y_k & x_{k,1} & \cdots & x_{k,n}\end{bmatrix}$$

The variables in x are supposed to be roughly independent (I can do a PCA to verify that, what would be the implications of an non-identity diagonalized matrix?), and I want to quantify which variables in x explain y the best.

The idea is to look at the principal axes in the PCA of Y, do you think this is the right way? How do I interpret the results then?

Answer 1

You can use PCA to verify independence of Xs. In this case if all eigenvalues from PCA analysis are of the same order the variables are independent if not (e.g. the ratio of th highest eigenvalue to the lowest has several zeros), the variables are probably lie on some lower dimensional hyperplane.

The next part concerns Ys. I don't think PCA is appropriate for the problem you have. One brute way you can employ to solve it is the following:

1. Choose all possible combinations of Xs.
2. For each combination perform leave-one-out cross validation analysis, i.e.
   • Take one combination of Xs
   • Construct regression model (e.g. multivariate linear or polynomial) using all observations but the one
   • Calculate prediction error on the observation excluded on the previous step
   • Do last two steps for each observation in the sample and calculate mean prediction error
   • Choose combination with lowest prediction error

Also take a look at this article: Sensitivity Analysis. It is exactly the problem you want to solve. Good luck :)