I'm going through Rethinking and Kruschke's Puppy book. After the examples I want to try myself with other data and have a problem.
What if (unlike the examples in the book and online) categorical variables have no hierarchy.
Thus numerical encoding feels wrong (maybe this is too traditional logistic regression thinking).
If I one-hot encode them, what kind of prior would I use? I am thinking Bernoulli but my Model diverges.
Thank you very much for any tips, hints and insights!
Based only on the Statistical Rethinking (2nd ed) book, it seems you are misunderstanding what the index variable (aka integer encoding) parametrization implies. I will clarify only this aspect of your question, as I think Tim's answer will be more in line with how to use dummy-coding instead.
You say:
What if (unlike the examples in the book and online) categorical variables have no hierarchy
But in page 155 the example used are female and male. He says explicitly:
Now "1" means female and "2" means male. No order is implied. These are just labels.
Even if dummy-coding is the norm in frequentist modeling (together with effects-coding in the ANOVA context), when we move to the Bayesian framework we introduce a new problem. Consider the same model in Chapter 5:
$\mu_i = \alpha + \beta_m m_i$
with $\mu_i$ being the average height for subject $i$, and $m_i$ being an indicator for whether a person is male or not.
Here the usual interpretation is that $\alpha$ denotes the average height for females, and $\beta_m$ is the difference in average height between males and females (so in this case, $\beta_m$ will be positive, as males are taller on average).
The problem comes when you put priors on the parameters, the example in Chapter 5 states:
$\alpha \sim \operatorname{Normal}(178, 20)$
$\beta_m \sim \operatorname{Normal}(0, 10)$
The issue described is that now you have two quantities that are defined by different number of parameters: female average height is determined by just $\alpha$, while male average height is determined by both $\alpha$ and $\beta_m$. Assume that female heights and male heights have the same variability, how would you encode this (and this is your question)? In this case, these priors indirectly state that the average height of males is more variable than for females, as it is a function of the variability of $\alpha$ and $\beta_m$. What an index variable does is instead of defining one $\alpha$ for one reference group, it defines an $\alpha$ per group: $\alpha_1$ and $\alpha_2$, which lets us set comparable priors for both with not much further thought (which is especially handy in more complex models).
In short, imagine that you have saved average heights for different groups in different boxes that have different names. An index variable is just a list of names used so you can retrieve the height from the appropriate box for that group. The boxes have no hierarchy whatsoever.
Answering given the clarification in the comments.
Bayesian logistic regression model is
$$\begin{align} p_i &= g^{-1}(\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k) \\ y_i &\sim \mathsf{Bernoulli}(p_i) \end{align}$$
with some priors for the $\beta_1,\beta_2,\dots,\beta_k$ parameters. $\beta_i$ are parameters of the model, and $X_i$ are the features. We are estimating the parameters, hence in the Bayesian setting, we are picking priors for them. We are not estimating the data, because the data was observed.
It doesn't matter that much $X_i$'s are binary or not, you pick the priors for the parameters, given what you expect a priori for the parameters to be. Now correcting what I just said, it is not exactly that you should not care at all what $X_i$'s are. Notice that in the Bayesian setting using something like Gaussian priors corresponds to using $\ell_2$ regularization, while using Laplace priors to $\ell_1$ regularization. In Ridge regression or Lasso, it is usually recommended to scale all the features because there is a single regularization parameter, i.e. we use the same prior for each parameter. For the same reason, using the same prior for parameters corresponding to different features that are differently scaled may accidentally bias your results. Usually, binary features would have different scales than many other features, so their parameters may need different priors, but in the same sense, each parameter needs an individual prior. So there is nothing special about binary features.
You may also find interesting the Gelman's et al paper The prior can generally only be understood in the context of the likelihood that discusses how our knowledge about the data influence the modeling decisions.
