3
$\begingroup$

I am running a OLS regression and to identify factors driving online sales of a product. This analysis (conducted only for inference) is run on different countries with the same variables included for each country, whether those variables are collinear or not. In short, some variables need to be included in the analysis despite sometimes showing a high VIF or p-value (above 0.05) for some countries.

The obvious choice to understand how variables driving sales is to look at coefficients. However, it is difficult to interpret these coefficients. One reason is that the variables are by nature not easy to compare. For example, some of independent variables are website visits, pickup sales (where an order is placed and collected in-store), product search (how many times the product was searched on google). Another reason is that the variables are standardized (because their units of measurement are completely different), so I can only say 'A change in standardized independent variable by one standard deviation will cause dependent variable to change by value of coefficient'.

Keeping this in mind I also ran dominance analysis, which calculates the individual contribution of variables in explaining the variance of online sales. This is especially useful in the presence of multicollinearity ("Because dominance is roughly determined based on which predictors explain the most variance, even when other predictors explain some of the same variance, it tends to de-emphasize redundant predictors when multicollinearity is present" Source).

While the definition of dominance analysis is straightfowrward, I am struggling to relate this to coefficients of variables. For example, if a variable has a high VIF or p-value, why could it show up as an important variable with dominance analysis? Is there a reason why I should trust one method over the other? How can I use relative variable importance to explain drivers of online sales?

$\endgroup$
1
  • $\begingroup$ Understanding relative importance is very useful information and something that ML algorithms haven't been able to address. The quick answer is that, unless a set of predictors have been standardized, regression coefficients are expressed in the units of x, i.e., a one unit change in x results in beta change in y. Gromping's article reviews the waterfront of the many methods for understanding relative importance, dominance analysis provides a related but different approach. Collinearity complicates any and all answers, bootstrapping predictors as in Breiman's random forest may help. $\endgroup$ Apr 22, 2022 at 18:12

1 Answer 1

3
$\begingroup$

As you note, model coefficients in a linear regression will tell you about change in predicted outcome for a one unit change in the predictor. Coefficients, standardized or not, then relate mostly to the the predicted levels/mean of the outcome.

By contrast, general dominance statistics (assume that's what is being interpreted) produces a partitioning of the $R^2$ which reflects all of the values of the predictor as translated into values of the predicted outcome. Thus, general dominance statistics reflect all of the predictor's variance at once--not one unit of change at a time. General dominance statistics then relate mostly to the predicted spread/variation of the outcome.

These statistics then answer two different questions (though they can sometimes give similar answers). If importance for your research question has more to do with change to levels/mean values given change on predictors, coefficients are to be preferred. This might be the case for an evaluation of a specific intervention.

If importance for your research question has more to do with total impact of a predictor across its entire range, dominance is to be preferred. This is usually the case in exploratory research when there may not be specific expectations about effects. It seems, given your description, this might be the situation you are in and that dominance statistics might be preferable.

For example, if a variable has a high VIF or p-value, why could it show up as an important variable with dominance analysis?

To this question, it is worth noting that dominance analysis is not very good at model selection and will ascribe non-0 shares of a fit statistic like the $R^2$ to predictors that have true non-0 effects when they are correlated with other variables that do (e.g., Budescu, 1993; Grömping, 2007). This is generally why those that have large p-values or VIF values would receive a potentially large dominance statistic when other statistics suggest they should not. That is, they should have been filtered in a previous, model selection, stage.


References

Budescu, D. V. (1993). Dominance analysis: a new approach to the problem of relative importance of predictors in multiple regression. Psychological bulletin, 114(3), 542.

Grömping, U. (2007). Estimators of relative importance in linear regression based on variance decomposition. The American Statistician, 61(2), 139-147.


Update

Responses to comment.

Could I please ask you to elaborate on 'specific expectations about effects'?

In short, does the researcher/analyst have a hypothesis about the direction and magnitude of the effect of a specific variable. For example, in experimental research this is often the case and in such cases understanding differences in levels/effect size is most valuable.

... how I could use 'total impact of a predictor' to explain drivers of online sales, and how it is different from 'unit change in predictor variable'

Will illustrate with an example:

data(mtcars)

> lm(mpg ~ vs + cyl + carb, data = mtcars |> datawizard::standardize()) |> coef() |> round(digits = 3)
(Intercept)          vs         cyl        carb 
      0.000      -0.149      -0.883      -0.170 

In the above, vs is a binary variable, cyl and carb are both categorical taking on 3 or 4 levels. All are standardized like the original example to facilitate comparison.

In this example, a one standard deviation change in vs results in a -.149 standard deviation change in mpg.

This information is useful as it tells the researcher "how" to get higher or lower values on the outcome (or explains why they got that way). If understanding how levels of the outcome got the way they are coefficients are most useful.

What coefficients do not do, is show how the variable actually behaves when applied to the levels of the predictors in the data.

For example, if the vs coefficient is applied to the values observed in the data, we see that its one standard deviation change concept is meaningless as it is a binary variable that takes on values of 0 and 1. It's standard deviation is ~.5 and there are no possible values that could occur at .5 of vs. So, all changes in the data are effectively at 2 standard deviations. Similar cases apply to carb and cyl. Those differences as applied to data are lost with coefficients as that's not what they are meant to convey.

By contrast, if we take all the predicted values in the data and correlate those with the outcome (and square it to obtain an $R^2$), we are evaluating how the values of the predictors "have translated" the coefficients into predicted values. That vs is a binary variable is necessarily incorporated into the $R^2$ as it is based on predicted values from the data. Dominance analysis is then an approach to ascribe components of the $R^2$ to the different variables to show their impact such as the example below:

> domir::domin(mpg ~ vs + cyl + carb, lm, list(performance::r2, "R2"), data = mtcars, complete = F, conditional = F)
Overall Fit Statistic:      0.7475552 

General Dominance Statistics:
     General Dominance Standardized Ranks
vs           0.1799347    0.2406975     2
cyl          0.4502766    0.6023322     1
carb         0.1173440    0.1569703     3

The ~.1799 value that vs obtains in the dominance analysis suggests that, on average, that is the $R^2$ using predicted values from this model. That value, again, accommodates the levels of the predictor as observed in the data and shows what it does when you take predicted values and correlate them with the outcome--hence, reflects the total effect of the variable across its observed range in the data.

This is useful as it shows the impact of the predictor as applied to the data, not abstracted from it as the coefficient is. Again, the two different methods supply different information about the model and data.

$\endgroup$
5
  • 1
    $\begingroup$ Please see stats.meta.stackexchange.com/questions/6304/my-upvoting-policy, when you find a question sufficiently clear to write an answer, consider to upvote the question! $\endgroup$ Apr 21, 2022 at 14:59
  • $\begingroup$ Thank you for the great answer. Could I please ask you to elaborate on 'specific expectations about effects'? I am trying to understand why dominance analysis could be useful and how I could use 'total impact of a predictor' to explain drivers of online sales, and how it is different from 'unit change in predictor variable'. Sorry if this is a redundant question. $\endgroup$ Apr 22, 2022 at 12:55
  • 1
    $\begingroup$ @TheloniousMonk, Have made an update to the post above to respond to the comments $\endgroup$
    – jluchman
    Apr 22, 2022 at 17:55
  • $\begingroup$ Thank you, its very clear now. My only confusion is now with dominance analysis not being a good way of model selection, and that variables with high p or VIF values should be removed beforehand. As dominance analysis typically is a way to support interpretation in the presence of multicollinearity (Source: frontiersin.org/articles/10.3389/fpsyg.2012.00044/full), it seems counter intuitive that I should remove multicollinear variables before using DA. Could you perhaps add some explanation as to why or refer me to the right literature? $\endgroup$ Apr 28, 2022 at 15:57
  • 1
    $\begingroup$ What is key to keep in mind is that, unless predictor variables are completely uncorrelated, a dominance approach will be useful in decomposing a fit metric like $R^2$. There will always be ambiguity about how to ascribe $R^2$ when predictors are correlated; dominance analysis provides a conceptually attractive method for ascribing the $R^2$ to them despite their correlation. Model selection approaches will not necessarily remove correlated predictors - the problem of model selection here is more on misspecification (discussed in the linked article as well in the last paragraph on DA). $\endgroup$
    – jluchman
    Apr 28, 2022 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.