# Get estimates about 'variable importance' across a large number of variables and their permutations

My client would like to know which, of many different kinds of product dimensions, are most important for perceived quality. Some of the dimensions are as follows: price, material, country of origin, whether or not your colleagues/friends have the product, and so on. Most of these dimensions include many levels (price, for example, can be from $$1 -$$1000; material can include steel, plastic, and so on; country of origin includes 10 possible countries).

We want to be able to answer general questions like: which of these factors are most important in predicting customer appraisals of quality? My client has created a survey in which a random sample of, say, three of these variables are shown to a customer (e.g. a 100 dollar, plastic, widget from China; a 500 dollar metal drum from Cambodia, etc.), and the customer rates how high they perceive the quality to be.

Coming from an econometric background, it's not clear how to answer this kind of question using my normal tools. There seems to be an enormous number of combinations of variables, and the potential for countless interaction effects seems overwhelming to interpret.

I've come across literature on conjoint analysis and taguchi methods, which seem relevant, but the articles I've found seem to broadly describe design principles, rather than the mechanics of the analytical strategy.

Random forests with variable importance seems promising, but it's not clear how to recover regression-esque effect sizes from forests, nor is it clear how to get a sense of which interactions are most relevant.

Perhaps some kind of Lasso regression? Would I fully specify all the interactions, and run a Lasso procedure? I'm worried it may select non-sensical interactions.

Apologies if the question is poorly specified. I'd like to be able to say, "If your product is coming from China, characteristics X, Y, Z are most important. If you are selling a plastic tool, characteristics A, B, C, are most important."

• As you already mentioned, I think conjoint analysis is usually used for this kind of thing. You say "it's not clear how to actually design and implement a study on this topic using those methods." Maybe you could clarify what about it is unclear, or why you think your case is exceptional, etc.? Jul 19, 2019 at 1:08
• What I don't understand is that conjoint analysis seems to more often describe the nature of the design rather than the right analytical approach. What is the exact model that should be used? Jul 19, 2019 at 1:14
• Not the exact scenario, but a similar scenario i worked is, input has some 100+ variables and the output is a vector of 5 values(numeric). So the client wanted to know,which of the input variables affect the output values. So we trained a neural network to predict the output values from the input values. Based on the gradient descent, the order of importance of these variables are found, and that tells are which are more influencing factors. Jul 19, 2019 at 6:15
• One idea: Use information on what determines perceptions of quality to constrain the dimensionality of the problem at the level of interactions. Wanting a straightforward regression-like interpretation is another way of saying the insight from the model regarding a variable cannot depend on too many other features in complicated ways to be actionable. If you're worried about non-sensical interactions, don't exhaustively search the space of non-sensical interactions. Model the interactions that could plausibly matter. Yes, this is a judgement call, but there are no assumption-free alternatives. Jul 20, 2019 at 12:13

In my field (political science) conjoint experiments are very common. Typically the data come in the form of a forced-choice comparison — i.e., survey respondents are shown a series of comparisons between product profiles in which features of the products are randomized, and then respondents choose which product they prefer.

In terms of analysis, a paper by Hainmueller, Hopkins, and Yamamoto shows that you can estimate a quantity called the average marginal component effect (AMCE) by a linear regression (using OLS) of the form $$Y_{ij} = a + \beta_1 X_{1ij} + \beta_2 X_{2ij} + \cdots + \epsilon_{ij},$$ where $$Y_{ij}$$ is an indicator for whether respondent $$i$$ chose profile $$j$$ when they had the option, and the $$X$$'s are vectors of indicator variables for the features. The AMCE is the average change in probability of choosing a product that has that specific feature, relative to the baseline, marginalizing over the distribution of other features. In other words, it averages over all possible interactions effects. This is a very simple estimator to implement since it's just OLS.

If you want to know how the importance of specific factors varies as a function of other factors, you could subset based on the other factor (e.g. whether the product was made in China) and re-estimate the regression above, or you could simply include interaction indicators. As you note, though, you can't estimate all possible interactions using OLS, since the number of combinations very quickly exceeds the number of respondents. Your idea to use LASSO makes sense to me. To see another estimator for this problem that has some nice properties, you could take a look at this paper by Egami and Imai.

• This is a superb answer. To summarize: the AMCE implicitly includes the average of all interaction effects (and thus does not need to be specified explicitly). However, it doesn't tell you what any given variable specifically interacts with, correct? For that an idea would be to implement a LASSO regression on a fully saturated model. Do you have any intuitions on using random forest regressions and computing interaction strengths via an H-statistic? (e.g. christophm.github.io/interpretable-ml-book/interaction.html) Jul 26, 2019 at 0:15
• Yes, the AMCE averages over the joint distribution of the other variables -- which will implicitly account for interaction effects. But you are also correct that it will not tell what those interaction effects are, nor whether they are important. I'm not too familiar with the H-statistic in the random forest case, sorry. Jul 26, 2019 at 4:17

Be cautious - most analysts do not use the bootstrap to get confidence intervals for variable importance measures, but when they do they are usually disappointed. The data, unless massive, do not contain sufficient information to tell you reliably which elements of the data are predictive, and do not have sufficient information for telling you how important each potential predictor is. I expand on this in Chapter 20 of BBR. See also this for a discussion of measures of added information.

• I think we have to be careful throwing around a term like “massive” here. I think I’ve seen you say something similar for split sample tests and then argue that after 20,000 records it’s probably okay. 20,000 to me is nowhere near massive. In fact with the data I use I would consider 20,000 to be quite small. I do see that in a clinical trial 20,000 is quite massive so it really depends on what type of data you are using. Jul 20, 2019 at 22:55
• Agreed. Most of my datasets involve hundreds of thousands of observations... I was thinking a "massive" dataset was in the millions. Jul 21, 2019 at 2:59

There are a whole host of options to determine variable importance. As you mentioned random forest has a variable importance metric built in. R package DALEX also has a model agnostic approach similar to the random forest importance but is model agnostic. When you say these don’t have effect sizes I would disagree and say they are quite interpretable. I would also suggest information theoretic methods such as information gain, though this assumes conditional independence between variables. Lastly you could look at the relief algorithm which to some extent considers dependence between predictors and works with all variable types

• Thanks for your answer. I'm familiar with DALEX and feature importance. The issue with these is that feature importance (depending on the kind) is interpreted in terms of the increase in mean-squared error if you randomly permute the variable in question. That doesn't have the kind of straight-forward interpretation as a coefficient in a regression equation, nor does it lend itself to significance testing. We want to be able to say "when you increase the price by X, it is associated with a decrease in Y price" I haven't heard of "relief algorithms", but they look promising. Jul 20, 2019 at 2:03
• Well relief and the information theory algorithms won’t give you that either. But I’m not sure something will aside from just running a regression if that’s what you want... Jul 20, 2019 at 2:14

Have you tried to look at Pareto-smoothed importance sampling (PSIS) by Vehtari, Gelman and Gabry? Here's a fast recap: "Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights". Look at the examples in the paper. It should be a good starting point. If it seems sounding to you there's an R package called loo` to work on these concepts.

• This is only for Bayesian models, and has nothing to do with how important a covariate is in prediction but rather for estimating out of sample error. Jul 20, 2019 at 20:47
• Yes, you're right here's the right paper to look at. link.springer.com/content/pdf/10.1007%2Fs11222-016-9649-y.pdf And definitely Yes, it is Bayesian statistic: is bayesian something to avoid? Jul 20, 2019 at 21:10
• Not necessarily, but the loo package wasn’t designed to solve the types of problems OP wants to solve. Jul 20, 2019 at 21:19