4
$\begingroup$

Let's say we are building a regression model with one nominal predictor, which has three levels, let's say red, blue, and yellow.

I remember being taught that when we build the model, we use j - 1 predictors in our model, where j is the total levels in your nominal predictor. In the above example where j = 3, we will have:

Y = B0 + B1X1 + B2X2

Where X1 and X2 are coded [0, 1] for red and blue respectively. If we want to see the effect of yellow on Y, it would just be B0.

I am reading a text on Bayesian stats where the author is building a model with a metric response and nominal predictor. It seems like in Bayesian stats, we end up having j predictors instead of j-1.

enter image description here enter image description here

Is my understanding correct? Depending on your approach, you may have a different number of predictors. Why can we do this in Bayesian and not have the issue of multicollinearity? What am I missing here?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The main reason you should keep all your dummies, not leaving one out when using bayes modeling, is that this simplifies the prior. This applies when using a proper, (weakly) informative prior. Such a prior functions as a kind of regularization, so this answer applies. Also note that when using a (proper) prior, the singular matrix issue that arises when keeping all dummies and using least squares, don't arise.

You could leave out one dummy and transform the prior, but there is nothing to gain, and ease of interpretation to loose.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks. This is still a bit over my head but I will have to look into this. I assumed it had something to do with LS estimation in traditional stats or the fact we force all the B1 coefficients to sum to 0 when doing Bayesian stats. $\endgroup$
    – confused
    Dec 10, 2019 at 15:29
  • $\begingroup$ Hello, I am reading my books/notes again and I am unable to find if we include B0 (the intercept) in the constraint of summing up all of the coefficients. Or does the constraint only apply to the B1 coefficients (the coefficients in front of the dummy variables). I am referring to Bayesian stats where we need the constraint of the coefficients summing to 0. $\endgroup$
    – confused
    Jan 1, 2020 at 18:53
  • 1
    $\begingroup$ Why do you need a restriction of some coefficients summing to zero? But anyhow, that would not include the intercept. $\endgroup$
    – kjetil b halvorsen ♦
    Jan 1, 2020 at 20:06
  • $\begingroup$ I figured it out. In Bayesian, all slope coefficients sum to 0. The intercept coefficient is not included in the sum. It's actually pretty ingenious how they force that constraint. Thanks! $\endgroup$
    – confused
    Jan 2, 2020 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.