Is there a way to determine which variable has more influence on dependent variable when your variables are highly correlated between themselves ? Most of the ways which deals with multicollinearity helps to improve prediction, like dropping variables who have high Variance inflation factor or combining variables who have high collinearity into single predictor, but they do not answer question which variable is more important.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ My approach is to iteratively bootstrap or jacknife a series of random samples, first pruning out those variables that are highly nonsignificant. Once the remaining set of variables has more or less stabilized at a p-value around p<=.15 or less, I focus on the issues wrt collinearity. The VIFs and eigenvalue indexes are good metrics at this point. By identifying variables (typically in pairs) that are collinear, prune away that variable with a lower t-value and is wrong-signed based on comparison of the parameter with a pairwise correlation. Eventually, a stable and noncollinear set emerges. $\endgroup$– user78229Commented Mar 15, 2017 at 15:31
-
$\begingroup$ The broader category your question falls into is variable-selection or feature-selection. There are multiple approaches to this type of problem. $\endgroup$– Matthew GunnCommented Mar 17, 2017 at 15:55
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Assuming you're doing OLS ($Y=X\beta_{ols}+\epsilon$), here's one approach using principal component analysis (PCA) that may provide some insight:
1) Demean and divide each of your independent variables by their standard deviation - this will allow easier interpretation later on.
2) Perform PCA on your $k$ number of independent variables, with $n$ observations. This will decompose them into scores, an $n$ by $k$ matrix $S$, and a $k$ by $k$ square transformation matrix $T$: $$ X=ST $$ The scores have the property that each column is completely uncorrelated with every other column, so that $cor(s_j,s_i)=0$ for all values of $j$ and $i$, and the transformation matrix $T$ will have the property of being invertable. Furthermore, the first column of $S$, lets say $s_1$, will be the most important in terms of explaining the commonalities between columns of $X$ (this is because the first column is associated with the largest eigenvalue).
3) Estimate your model using the scores: $$ Y=S\beta_{pca}+\epsilon $$ The coefficients are not immediately interpretable, so next is where the trick comes in.
4) Look at the $j^{th}$ column of $T^{-1}$ corresponding to the $\beta_{pca,j}$ with largest t-statistic - this will tell you what relationships within $X$ are most important. The reason is made clear after re-writing identities above: $$ \hat{Y}=X\beta_{ols}=(S)\beta_{pca}=(XT^{-1})\beta_{pca}$$so$$ \beta_{ols}=T^{-1}\beta_{pca} $$ Because of the first step, the magnitudes of the elements in $T^{-1}$ are directly comparable, and each column denotes a linear combination of your $X$s. You're interested in what linear combinations are most relevant, so that, for example, if one variable is the most important, the $j^{th}$ column of $T^{-1}$ will have a an element in it much larger than the rest of the elements, and will correspond to the "most important" variable in $X$.
-
$\begingroup$ I have one question. In step 3 what I need is to look for the best model with these scores ? $\endgroup$– ExtriaCommented Mar 16, 2017 at 13:52
-
$\begingroup$ For model selection, you're right that you don't need to go beyond step 3. Because the columns are orthogonal, if all the p-values on $\beta_{pca}$ are significant, then all of the variables included in the pca significantly contribute predictive power. If there is at least one p-value that is not significant, then you should consider a different set of variables (a subset of the original), perhaps with different transformations (e.g. spreads) - however, there is no way, by just looking at the p-values, to determine which variables in $X$ are not contributing in this case. $\endgroup$ Commented Mar 16, 2017 at 14:15