The Gist

What are some methods of quantifying "interaction" among features for a model? I will give two short motivating examples to clarify what I mean by "interaction".

Motivating Examples

  1. With a linear model, it is likely that a few features will dominate some of the other features to some degree. This does not necessarily mean that those other features are bad, although they will probably be given a lower weight and would have performed better in the absence of the dominating features (or if the dominating features had been given a lower weight).
  2. With a tree ensemble model, it is likely that a more important feature will be split near the top of the tree and that some other dominated feature will get its chance to shine after that split or in some split that does contain the dominating feature.


It would be useful to be able to represent relationships between features in some humanly interpretable way. I would especially want to see relationships between some feature F1 and some F2 in which F2 shines in the absence of F1 or contributes a lot more information towards predicting the outcome when F1 has been split (as in a decision tree).

This information might be represented as a list of triples (F2, F1, score) sorted in descending order by score, where score might represent how much more information F2 contributes towards predicting the outcome in the absence of F1 (or after F1 has been split in the case of a tree ensemble).

It's possible that the linear and tree ensemble may represent separate cases (absence of F1 vs splitting F1). It is also possible that there may be multiple useful metrics for each case. I do not necessarily want to select or remove features, but merely to see how they interact. (In my particular case, I have about 100 features.)


2 Answers 2


Good questions even though no single response can address all of your concerns. This will be, at best, a partial answer. It will also be from an applied -- not theoretical level -- it is practical advice. It sounds like you're thinking only of two-way interactions but, obviously, three-way and higher interactions are possible. In fact, one can imagine extending the interactions up to a "fully saturated" limit that would exhaust the degrees of freedom. So, let's limit the discussion to two-way interactions which are easily generalizable to higher ones.

Let's also mention that statistical theory recommends (or prescribes) that the appropriate main effects be retained in any model containing interactions with those effects. This recommendation isn't always followed or observed, but it needs to be noted -- do with it what you will. The more important thing to note is that the interpretation of the model parameters shifts from one of being conditional on the mean values of the features when there are only main effects (i.e., for models that are evaluating the mean as in OLS regression. This would change with differing functional forms such as quantile regression) to an interpretation in the presence of interactions that is conditional on the features being evaluated at zero. In applied terms, this means that, before taking the interaction, the features should be normalized to a mean of zero. Regrettably, none of the points made in this paragraph are without controversy and many statisticians can and will disagree with them. I can provide references supporting these statements if anyone is interested.

That said, the first approach to incorporating interactions can be described as a "global," "brute force" or "shotgun" approach insofar as all possible permutations of (two-way) interactions are evaluated. I say permutations and not combinations since interactions are symmetric -- it doesn't matter if the interaction is F1 with F2 or F2 with F1. Obviously, with more than a few features this approach would quickly result in an unmanageably large number of interactions. Regardless, it is possible and has been done.

A second, somewhat more reasonable approach is to use subject matter expert (SME) knowledge to make judgments as to which interactions are interesting or useful to model. This has the advantage of immediately reducing the number of interactions to a manageable size as well as minimizing time wasted on potentially fruitless exploratory analysis. On the other hand, this approach has the disadvantage that it presumes the SME knows everything there is to know about the data (in my experience this is rarely, if ever, the case), thereby precluding the possibility of serendipitously uncovering new relationships.

There is a third approach that is less reliant on SMEs and more empirically driven. This method assumes, as you have noted, that a relative handful of features are explanatorily important, i.e., they rank towards the top of relative variable importance. By focusing interactions on that subset of features, it becomes possible to derive a more manageably sized set of interactions, as long as there aren't too many important, candidate features.

So, how might one address your concern with pulling together an "ensemble" of metrics providing statistically useful information regarding the relationships between and among the features and their interactions. For me, this is entirely a function of the analyst's appetite, patience and time available for playing around with the results from an ensemble of differently specified models. A simple example of how this could work is OLS ANOVA with two features and one two-way interaction. Depending on the software, it is possible to specify that the sums of squares of the features and interaction be treated independently in entering the model -- some packages describe this as SS3 (for sums of squares type 3). One could also try SS1, SS2 or SS4 which invoke different orderings of the independent variables. By playing around with the permutations of these orderings, the "humanly interpretable" information can be deduced...that is, up to the limits of human cognition in juggling a large number of differing metrics. Ulrike Groemping has done the most work in reviewing and explaining the different statistical approaches to this kind of model evaluation of relative variable importance. She also has an R module which does a computationally intensive version of this called RELAIMPO. Check her papers out.

My personal solution is different from hers. Imagine you have summary performance statistics across a set of different model specifications returning metrics such as score, r-square, AIC, different specifications of SS types, and so on. Making sure they are all in a consistent unit of analysis would be a necessary step. Once that's done and with enough observed summary values, a PCA of these metrics becomes feasible. By scoring the input data based on the first PCA, then a summary ranking of the units under analysis is possible. I'm not sure I'm explaining this well enough but, having gotten it to work, it can be a useful tool for information reduction particularly in the case when one has a set of competing summary metrics. Pushback from other CV participants is expected here.


A linear model wont in the general case make your features sparse, unless you eg choose a loss function that encourages sparsity, such as L1/Lasso.

MSE loss on a linear model will not encourage sparsity, or one feature to dominate another, in general, for example.

In the case that you are using Lasso/L1 loss, then some of your features are likely to be driven to zero. You can then use eg the correlation matrix, $X^TX$, to look at correlations between the non-zero features and the zero features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.