I am trying to interpret the regression coefficients of a covariate in a Bayesian linear regression problem. More specifically, I am trying to determine if the regression coefficient have an important effect on the prediction of the response variable. A discsussion of this can be found here in a Bayesian context (see section 5.2.3).

From my understanding, when the posterior distribution of the estimated regression coefficient is away form the zero, its suggests an important contribution of the covariate to the prediction of the response variable.

Here is the posterior distribution from regression coefficient:

enter image description here

The posterior mean of this distribution is 0.018 and the 95% credible interval is -0.01 and 0.045.

My question is: Since the mean is 0.018 and away from zero, can I say this regression coefficient had an important effect on the prediction/estimation of the response variable ?


Can I say that: Since the zero value lies between the 95% credible interval of the covariate's posterior distribution, then this regression coefficinet DOES NOT have an important effect on the response variable ?

My issue is that I am not sure if to use the posterior mean or the posterior credible interval to determine the effect of a regression coefficient on the dependent variable, i.e., I am not sure which property (i.e., mean or Cred Int.) of the posterior distribution to assess this impact.

  • $\begingroup$ Bayesian inference as such can't tell you if a coefficient is substantively important. That's something you have to decide as the researcher. The posterior is what it is. If you're asking because you have variable selection in mind, there are Bayesian approaches to that: en.wikipedia.org/wiki/Spike-and-slab_variable_selection $\endgroup$ – John Zito Aug 17 '18 at 19:54
  • $\begingroup$ please see edit to my post providing a link to discussion on importance of variables $\endgroup$ – user121 Aug 17 '18 at 20:12
  • $\begingroup$ I don't think that text is giving good advice about how to evaluate "importance" in a Bayesian framework. It's basically describing rules of thumb for how to use the output of a Bayesian analysis as if it were the output of a frequentist analysis, which misses the point. At the end of the day, "importance" won't be adjudicated by making ad hoc judgements about how far the posterior mean/median is from 0, or whether some (which?) credible region contains 0. $\endgroup$ – John Zito Aug 17 '18 at 23:12
  • $\begingroup$ A better way to go would be to consider two models: the one that includes the covariate, and the one that does not. You can compute Bayes factors and do proper Bayesian model comparison to directly address the question "Does including this covariate matter for predicting the response?" $\endgroup$ – John Zito Aug 17 '18 at 23:15

Unfortunately, it's not possible to deduce whether a covariate has an important effect on the target variable just from the coefficient, for some imprecise meaning of the word "important."

Consider, for example, the following model:

$$y_t = \beta_1x_t + \beta_2z_t + e_t$$

with $\beta_1 = 1$, $\beta_2 = 10$, and $x$, $z$ and $e$ independent. Can we conclude that $z$ has an important effect on $y$? No, as we need to the distribution of values of $x$, $z$, and $e$. Let us say that $x \sim \text{N}(0,1)$, $e \sim \text{N}(0,1)$, and $z \sim \text{N}(0, 0.000001)$. Then the total variability of $y$ is:

$$\sigma^2_y = \beta_1^2\sigma^2_x + \beta_2^2\sigma^2_z + \sigma^2_e = 1 + 0.0001 + 1 = 2.0001$$

of which $0.0001$ is due to $z$, if I've counted my zeroes correctly. This would not typically be considered "important", although with enough data the parameter estimate of $\beta_2$ could well be significantly different to 0.

  • $\begingroup$ please see the link here. Specifically Section 5.2.3 and bottom of pg 161 where an example is discussed on importance of coefficients. let me know your thoughts. appreciated. $\endgroup$ – user121 Aug 17 '18 at 18:54
  • 1
    $\begingroup$ He is using the word "important" in a sense similar to "statistically significant" in a frequentist analysis, rather than in the sense of "of practical importance, worth worrying about when modeling." Effects can be "statistically important" while simultaneously "not practically important", for, again, imprecise meanings of the two phrases, but you get the idea. $\endgroup$ – jbowman Aug 17 '18 at 19:00
  • $\begingroup$ from my post, would you say my covariate is statistically significant based on its posterior distribution? $\endgroup$ – user121 Aug 17 '18 at 19:03
  • $\begingroup$ To answer your question narrowly, no. More broadly, though, I think statistical significance as a concept does not mesh well with Bayesian statistics, although many, many people have tried to figure out how to do so. I admit I'm not up-to-date on all of these efforts, though. If I have a good a-priori reason for including a variable in a model, I tend to just leave it there, or remove it and see what happens to my out-of-sample predictive accuracy. In industrial work, though, you rarely have the time to do everything you'd like to do, so don't take me as much of a guide! $\endgroup$ – jbowman Aug 17 '18 at 19:12
  • $\begingroup$ appreciate your input. May you please expand as to why you said no on the statistical significance. $\endgroup$ – user121 Aug 17 '18 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.