2
$\begingroup$

I'm going through Rethinking and Kruschke's Puppy book. After the examples I want to try myself with other data and have a problem.

What if (unlike the examples in the book and online) categorical variables have no hierarchy.

Thus numerical encoding feels wrong (maybe this is too traditional logistic regression thinking).

If I one-hot encode them, what kind of prior would I use? I am thinking Bernoulli but my Model diverges.

Thank you very much for any tips, hints and insights!

$\endgroup$
10
  • $\begingroup$ Could you give more mathematical formula e.g., on one-hot encode? It seems that it is not a common term. $\endgroup$
    – TrungDung
    Apr 12, 2022 at 18:36
  • 2
    $\begingroup$ One-hot encoding == Dummy Variables $\endgroup$
    – Avraham
    Apr 12, 2022 at 18:50
  • 2
    $\begingroup$ Priors are not used for features but for parameters. Why parameters for binary features should have different distributions than other ones? $\endgroup$
    – Tim
    Apr 13, 2022 at 5:42
  • 1
    $\begingroup$ @Oliver the feature is data not a parameter, if it was a parameter you wouldn't need to estimate it because you know what the feature is. $\endgroup$
    – Tim
    Apr 13, 2022 at 7:58
  • 1
    $\begingroup$ No, features are not parameters. $\endgroup$
    – Tim
    Apr 13, 2022 at 8:32

2 Answers 2

2
$\begingroup$

Based only on the Statistical Rethinking (2nd ed) book, it seems you are misunderstanding what the index variable (aka integer encoding) parametrization implies. I will clarify only this aspect of your question, as I think Tim's answer will be more in line with how to use dummy-coding instead.

You say:

What if (unlike the examples in the book and online) categorical variables have no hierarchy

But in page 155 the example used are female and male. He says explicitly:

Now "1" means female and "2" means male. No order is implied. These are just labels.

The Bayesian problem with dummy-coding (aka one-hot encoding)

Even if dummy-coding is the norm in frequentist modeling (together with effects-coding in the ANOVA context), when we move to the Bayesian framework we introduce a new problem. Consider the same model in Chapter 5:

$\mu_i = \alpha + \beta_m m_i$

with $\mu_i$ being the average height for subject $i$, and $m_i$ being an indicator for whether a person is male or not.

Here the usual interpretation is that $\alpha$ denotes the average height for females, and $\beta_m$ is the difference in average height between males and females (so in this case, $\beta_m$ will be positive, as males are taller on average).

The problem comes when you put priors on the parameters, the example in Chapter 5 states:

$\alpha \sim \operatorname{Normal}(178, 20)$

$\beta_m \sim \operatorname{Normal}(0, 10)$

The issue described is that now you have two quantities that are defined by different number of parameters: female average height is determined by just $\alpha$, while male average height is determined by both $\alpha$ and $\beta_m$. Assume that female heights and male heights have the same variability, how would you encode this (and this is your question)? In this case, these priors indirectly state that the average height of males is more variable than for females, as it is a function of the variability of $\alpha$ and $\beta_m$. What an index variable does is instead of defining one $\alpha$ for one reference group, it defines an $\alpha$ per group: $\alpha_1$ and $\alpha_2$, which lets us set comparable priors for both with not much further thought (which is especially handy in more complex models).

In short, imagine that you have saved average heights for different groups in different boxes that have different names. An index variable is just a list of names used so you can retrieve the height from the appropriate box for that group. The boxes have no hierarchy whatsoever.

$\endgroup$
3
  • 1
    $\begingroup$ Thank you so much! I was going to bother Tim further with actually an height and weight example but I thought I rather re-read again. This actually answers my question and confusion. Thanks a lot to both of you! This was my question and you answered it. $\endgroup$
    – Oliver
    Apr 13, 2022 at 11:23
  • $\begingroup$ Is this basically hierarchical regression? Or is this different in some way? $\endgroup$ Sep 20, 2022 at 23:50
  • $\begingroup$ Where do you see the hierarchical aspect of it? $\endgroup$
    – Kuku
    Sep 21, 2022 at 8:47
1
$\begingroup$

Answering given the clarification in the comments.

Bayesian logistic regression model is

$$\begin{align} p_i &= g^{-1}(\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k) \\ y_i &\sim \mathsf{Bernoulli}(p_i) \end{align}$$

with some priors for the $\beta_1,\beta_2,\dots,\beta_k$ parameters. $\beta_i$ are parameters of the model, and $X_i$ are the features. We are estimating the parameters, hence in the Bayesian setting, we are picking priors for them. We are not estimating the data, because the data was observed.

It doesn't matter that much $X_i$'s are binary or not, you pick the priors for the parameters, given what you expect a priori for the parameters to be. Now correcting what I just said, it is not exactly that you should not care at all what $X_i$'s are. Notice that in the Bayesian setting using something like Gaussian priors corresponds to using $\ell_2$ regularization, while using Laplace priors to $\ell_1$ regularization. In Ridge regression or Lasso, it is usually recommended to scale all the features because there is a single regularization parameter, i.e. we use the same prior for each parameter. For the same reason, using the same prior for parameters corresponding to different features that are differently scaled may accidentally bias your results. Usually, binary features would have different scales than many other features, so their parameters may need different priors, but in the same sense, each parameter needs an individual prior. So there is nothing special about binary features.

You may also find interesting the Gelman's et al paper The prior can generally only be understood in the context of the likelihood that discusses how our knowledge about the data influence the modeling decisions.

$\endgroup$
3
  • $\begingroup$ If I understand correctly, and based on the framing of the problem in one of the referenced books, the OP's problem is how to set priors for the $\beta_i$ if there is a reference category in $\beta_0$, and we wish the priors across categories to be comparable. $\endgroup$
    – Kuku
    Apr 13, 2022 at 9:30
  • $\begingroup$ @Kuku it might be the case that I wrongly assumed that I understand the question, but my answer refers to what I understood as a core issue of the misunderstanding. $\endgroup$
    – Tim
    Apr 13, 2022 at 9:39
  • 1
    $\begingroup$ Thank you so much for writing this up, together with Kuku's answer this was extremely helpful. Thank you @Tim thank you @Kuku! $\endgroup$
    – Oliver
    Apr 13, 2022 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.