When doing logistic regression with data $D_N = \{(x_i, y_i)\}_i^N$ with $x_i \in \mathbf{X}^N$ (each data point has N features) and $y_i \in \mathbf{Y}$ being assigned output classes, in a Bayesian framework, we would assign prior distributions to the model parameters $\theta$, compute the posterior

$p(\theta|D_N) \approx p(\theta) L(y|x,\theta)$

and then sample from the posterior over $\theta$ to generate predictions on output class $y^*$ for a new observation $x^*$ given the model trained on data $D_N$ using the posterior predictive distribution

$p(y^*|x^*,D_N) = \int p(y^*|x^*,\theta) p(\theta | D_N) d \theta$.

I would like to know how to incorporate prior beliefs on output class $y$ given $x$ to influence the predictive distribution.

So far we only talk about setting priors on the model parameters. But I more often have available the correlations between data features and an output class (e.g. my input data on people $x$ has one feature "academic degree" for which I have priors on output classes 'wealthy' vs 'homeless').

I understand these priors are conditioned on features of my data, so I would either have to choose some, or average them. But in general I am lacking the term $p(y|x)$ independent of model parameters $\theta$.

What is the correct Bayesian way of including this?

Some ways I have seen so far of including prior information of output classes:

1) Into hyperparameters of the training algorithm in supervised learning. This is mostly in ways of introducing prior knowledge on the overall frequencies of classes, doing some kind of upsampling or sample-weighting in order to resolve class imbalance.

2) Into the parameters of the model. I am not quite sure how this is done, but I suspect that the prior predictive distribution is calibrated so that it resembles the expected output of classes before observing data. This is apparently analogue to the empirical Bayes approach: Translating our prior evidence on data into the parameter priors, in such a way that the prior predictive distribution resembles our beliefs.

These are nevertheless both indirect ways. There must be alternatives of including scores from previous prior probability distributions / scores into the output of a classifier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.