Is there a method similar to commonality analysis but can be applied beyond the scope of multiple regression? I got one dependent variable and six independent variables. All of them are continuous.
First, I built a linear regression model, but the R2 was only 0.22.
Then I tried to build a random forest model with the same data. The R2 increased to 0.78.
After comparing the results of both models on an independent test set, I think random forest model was not over-fitted. And I'm satisfied with the result. 
When I built a linear regression model, I could use commonality analysis
to decompose R2 into unique and common variance of independent variables. Now the question is which method should I use to decompose the R2 explained by the random forest model in a similar way?
My first thought was to build a full model and a model without some independent variable (Let's say x). The difference of R2 between these two models should be the part explained by x. However, it turned out some model with less independent variables had higher R2 than full model. So this is an invalid method. 
Could anyone give some suggestions on this? Any help would be appreciated. 
 A: Your tool of choice is most likely to be the variable importance (VI) measure. That is the loss of cross-validated model predictive performance if a given variable is permuted after training.
I did develop something quite similar commonality analysis for random forest. The remaining answer, was my thoughts on that.
Regular VI do not reveal any information whether variables are redundant or complimentary. A year ago, I defined, a two-way VI term. But I'm probably not the first to get that idea. Two-way VI can under favorable conditions (n.obs, n.var, etc.) describe variable redundancy and complementarity.
Twoway VI is defined as the change of regular VI of one variable (A) conditioned by the permutation of second variable (B).
 - If the two-way VI is positive, this points to the model relied more on A when missing B. This points to A is redundant to B.
 - If the two-way VI is negative, this points to the model could not make as much use of A when missing B. This points to A being complimentary to B.
Two-way VI can be computed with and without retraining of model.


*

*If retraining is performed after permuting B, redundant A will improve its oneway VI. If retraining is not performed after permuting B, redundant A will only improve oneway VI slightly, as the model have not adapted to rely more on A.

*If retraining is not performed after permuting B, complimentary A will decrease its oneway VI. The model have not adapted to replace B to form a meaningful interaction effect with A. If retraining is performed after permuting B, complimentary A may only slightly decrease oneway VI, as the model can replace B to make use of A.
Retraining is best to spot variables being redundant. No retraining is best to show variables being complimentary, due to a learned interaction effect.
Two variables can be both complimentary and redundant to each other.
The drawbacks of twoway-VI:


*

*It is not three-, four- or fiveway. Thus redundancy and complementarity can be masked by variable C and D also being either redundant or complimentary. It would take too much time to computing higher orders VI and too uncertain.

*Only strong redundant/complimentary variables were indentified, even with many simulation reruns to reduce the background noise.

*Variable importance is a global term generalizing the influence of a variable to the entire feature space. This may not be useful. A model may learn a interaction effect between variable in one part of feature space (making the variables locally complimentary), while in another place learn a weighted average of the variables (making the variables partly redundant). Local VI has been invented, but it would likey be too uncertain to combine with twoway-VI.


To better understand when a RF model fit make use of weighted averages of variables and when it make use of interactions effects I shamelessly recommend my own R package, forestFloor, described in this post.
A sketch on how to compute variable importance.
oneway-VI is computed by this sequence: Train model -> compute OOB.CV.1 -> Permute variable A -> compute OOB-CV.2
oneway-VI = OOB.CV.1-OOB.CV.2
twoway-VI.v1 is computed by this sequence: Train model -> compute OOB-CV.0 -> Permute variable A -> compute OOB-CV.1 -> Permute variable B -> compute OOB-CV.2
twoway-VI = (OOB.CV.0 - OOB.CV.1) - (OOB.CV.0 - OOB.CV.2) = OOB.CV2 - OOB.CV1
twoway-VI.v2 is computed by this sequence: Train model -> compute OOB-CV.0 -> Permute variable A -> compute OOB-CV.1 -> Reconstruct variable A -> Permute variable B -> retrain model -> compute OOB-CV.2 -> Permute variable A -> compute OOB.CV.3
twoway-VI.v2 = OOB.CV.0 - (OOB.CV.1) - (OOB.CV.2 - OOB-CV.3)
