I observe y as a function of x. y is a scalar, x is a vector of dimension n. I have k observations of y and x.

Let $$X = \begin{bmatrix}x_{1,1} & \cdots & x_{1,n}\\\vdots & \vdots & \vdots \\ x_{k,1} & \cdots & x_{k,n}\end{bmatrix}$$

$$Y = \begin{bmatrix}y_1 & x_{1,1} & \cdots & x_{1,n}\\\vdots &\vdots & \vdots & \vdots \\ y_k & x_{k,1} & \cdots & x_{k,n}\end{bmatrix}$$

The variables in x are supposed to be roughly independent (I can do a PCA to verify that, what would be the implications of an non-identity diagonalized matrix?), and I want to quantify which variables in x explain y the best.

The idea is to look at the principal axes in the PCA of Y, do you think this is the right way? How do I interpret the results then?


You can use PCA to verify independence of Xs. In this case if all eigenvalues from PCA analysis are of the same order the variables are independent if not (e.g. the ratio of th highest eigenvalue to the lowest has several zeros), the variables are probably lie on some lower dimensional hyperplane.

The next part concerns Ys. I don't think PCA is appropriate for the problem you have. One brute way you can employ to solve it is the following:

  1. Choose all possible combinations of Xs.
  2. For each combination perform leave-one-out cross validation analysis, i.e.
    • Take one combination of Xs
    • Construct regression model (e.g. multivariate linear or polynomial) using all observations but the one
    • Calculate prediction error on the observation excluded on the previous step
    • Do last two steps for each observation in the sample and calculate mean prediction error
    • Choose combination with lowest prediction error

Also take a look at this article: Sensitivity Analysis. It is exactly the problem you want to solve. Good luck :)

  • $\begingroup$ (+1) Nice answer. Welcome to CrossValidated, Paul! $\endgroup$ – MånsT Jul 13 '12 at 11:38
  • $\begingroup$ I had a look at the sensitivity analysis, yet I couldn't find if it could be used for unordered discrete variables (like labels). Also, if I want to rank my variables from the one which explains the most to the one explaining the least the output, should I use some uncertainty analysis? I wasn't able to find methods for that field... $\endgroup$ – Flavian Jul 14 '12 at 20:29
  • $\begingroup$ You mean that your Xs are labels (like 'hot', 'cold', etc.)? Then you need to somhow quantify them in order to perform futher analysis (convert them to reasonable scale, e.g. in this case temperature from -1 ['cold'] to +1 ['hot']). Also read here sensitivity-analysis.jrc.ec.europa.eu/methods/index.htm for some usefull methods $\endgroup$ – Paul Jul 15 '12 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.