1
$\begingroup$

I have experimental data. The data consists of simultaneous observations of some external variable $Y$ and multiple intrinsic variables $\vec{X}$. It is known that all $\vec{X}$ are strongly correlated to each other, in other words, the data is multicollinear to the extent where this cannot be ignored. The original research question is: which of $\vec{X}$ explain $Y$, and to what extent - a typical ANOVA use case. In view of multicollinearity, the refined research questions are as follows:

  1. What is the total explained variance $V_{tot}$ of $Y$ due to all $\vec{X}$ together?
  2. What is the explained variance $V_i$ of $Y$ due to each of $X_i$ individually, ignoring the other $\vec{X}$ values?
  3. What is the unique explained variance $V_i^{unq}$ of $Y$ due to each of $X_i$, that cannot be (linearly) explained by any other $X_i$ or their linear combination?

So the research question is to try to put robust lower and upper bound on explained variance due to each predictor, and then test for significance.

Answering the first two points is relatively simple:

  • for 1. we perform a basic linear regression, find the total explained variance and test using F-test (multi-way ANOVA)
  • for 2. we perform univariate linear regressions for each variable independently.

However, for 3. the solution is not that simple. While this is what I intuitively expected multi-way ANOVA to do, I recently found out that in case of severe multicollinearity, individual linear regression coefficients are essentially random, the related explained variances and p-values essentially random and not useful. Literature suggests that Ridge regression or Lasso are specifically designed to work with multicollinear data, but I do not think that they answer my research question (3). In my understanding, both regularization procedures would attempt to make the coefficients more robust by selecting the only the strongest correlated predictors out of all of the redundant predictors. So the regularization procedures attempt to make a guess at how the redundancy is distributed across the predictors. I don't want them to do that because I don't know in advance if their assumptions are correct. To the best of my understanding, the solution for (3) is the partial correlation between $Y$ and $X_i$ conditioned on all other $\vec{X}$.

Questions:

  1. Is it correct that multi-way ANOVA is not useful for strongly multicollinear data? Do I understand correctly that the standard way to obtain robust coefficient estimates is to introduce a regularization term? Does a regularization term guarantee robustness of coefficients to small perturbations in data? Is it scientifically acceptable that the reported explained variances and p-values would depend on the (somewhat arbitrary) choice of regularization parameter?
  2. Is Partial Correlation a good way to address my research goal (3)? Are there other ways?
  3. If I go for partial correlation, is it easy to obtain "unique explained variances" from the partial correlation coefficients? Or are they already reported by some standard implementations of the method?
$\endgroup$
1
$\begingroup$

This is a well-known phenomenon and has been studied thoroughly. As an aside, this is not an ANOVA setup unless the Xs are mutually exclusive indicator variables (in which case they would not be collinear).

The answer to the first question is trivial, and quantifying the total extent of association between X and Y is not hurt by collinearity. Question two is not very relevant. Question 3 is impossible to answer when there is strong collinearity. The sampling distributions of the regression coefficient estimates are degenerate and you'll get arbitrary answers. In general, partial $R^2$ is a good approach, but it doesn't help here.

If you can partition X into clusters such that two columns of X that are in different clusters have very low correlations, you can compute composite ("chunk") tests and associated partial $R^2$ for each cluster. I go through this in RMS.

$\endgroup$
8
  • $\begingroup$ Hi Frank. Thanks for the answer, I will have a look at your course details. I do not understand why goal (3) would not be possible to answer. In general, one can construct a likelihood function (or posterior in bayesian cases) over all possible multidimensional coefficient values, then find the marginal confidence/credibility intervals for each parameter, thus verifying if any parameter is non-zero with high confidence. Can you please elaborate? For me the question is not whether it is possible, but what is the most comfortable way to do it $\endgroup$ Oct 12 '21 at 12:01
  • $\begingroup$ The impossibility of answering that question is reflected in any of the following. (1) In a frequentist analysis, standard errors of partial effects upon adjustment for a competing variable is very large, signaling great uncertainty in estimating the partial effect. (2) In a Bayesian analysis the posterior distribution is degenerate and densities are wide. (3) If you bootstrap whatever partial effect measure you choose you'll see hugely wide confidence intervals for that measure. (4) If you get a new sample the measures will disagree with the initial partial effect measures. $\endgroup$ Oct 12 '21 at 12:09
  • $\begingroup$ If I understand your comment correctly, main problem is not so much in the theoretical justification of the approach, but in its data hunger. In other words, given infinite data, I will most likely be able to find robust partial correlation estimates, but for commonly available finite data sizes my variances will be so wide that all tests will end up insignificant. Is this correct? $\endgroup$ Oct 12 '21 at 12:13
  • 1
    $\begingroup$ I've made a simulation to disprove your point. And it seems I disproved my point instead :D. I have posted a much more focused question about partial correlation. I will return to this question once I have resolved my confusion about that question. Thank you for your feedback so far, it has been very useful $\endgroup$ Oct 14 '21 at 11:58
  • 1
    $\begingroup$ Simulations are invaluable. Like the bootstrap on one dataset, simulating multiple datasets frequently exposes instability. $\endgroup$ Oct 14 '21 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.