Dirichlet prior is an appropriate prior, and is the conjugate prior to a multinomial distribution. However, it seems a bit tricky to apply this to the output of a multinomial logistic regression, since such a regression has a softmax as the output, not a multinomial distribution. However, what we can do is sample from a multinomial, whose probabilities are given by the softmax.
If we draw this as a neural network model, it looks like:
We can easily sample from this, in the forwards direction. How to handle backwards direction? We can use the reparameterization trick, from Kingma's 'Auto-encoding variational Bayes' paper, https://arxiv.org/abs/1312.6114 , in other words, we model the multinomial draw as a deterministic mapping, given the input probability distribution, and a draw from a standard Gaussian random variable:
$$
x_{\text{out}} = g(\mathbf{\mathbf{x}_{\text{in}}}, \mathbf{\epsilon})
$$
where: $\mathbf{\epsilon} \sim \mathcal{N}(0, 1)$
So, our network becomes:
So, we can forward propagate mini-batches of data examples, draws from the standard normal distribution, and back-propagate through the network. This is fairly standard and widely used, eg the Kingma VAE paper above.
A slight nuance is, we are drawing discrete values from a multinomial distribution, but the VAE paper only handles the case of continuous real outputs. However, there is a recent paper, the Gumbel trick, https://casmls.github.io/general/2017/02/01/GumbelSoftmax.html , ie https://arxiv.org/pdf/1611.01144v1.pdf ,and https://arxiv.org/abs/1611.00712 , which allows draws from discrete multinomial papers.
The Gumbel trick formulae gives the following output distribution:
$$
p_{\alpha, \lambda}(x) = (n-1)! \lambda^{n-1} \prod_{k=1}^n \left(
\frac{\alpha_k x_k^{-\lambda-1}}
{\sum_{i=1}^n \alpha_i x_i^{-\lambda}}
\right)
$$
The $\alpha_k$ here are prior probabilities for the various categories, which you can tweak, to push your initial distribution towards how you think the distribution could be distributed initially.
Thus we have a model that:
- contains a multinomial logistic regression (the linear layer followed by the softmax)
- adds a multinomial sampling step at the end
- which includes a prior distribution over the probabilities
- can be trained, using Stochastic Gradient Descent, or similar
Edit:
So, the question asks:
"is it possible to apply this kind of technique when we have multiple predictions (and each prediction can be a softmax, like above) for a single sample (from an ensemble of learners)." (see comments below)
So: yes :). It is. Using something like multi-task learning, eg http://www.cs.cornell.edu/~caruana/mlj97.pdf and https://en.wikipedia.org/wiki/Multi-task_learning . Except multi-task learning has a single network, and multiple heads. We will have multiple networks, and a single head.
The 'head' comprises an extract layer, which handles 'mixing' between the nets. Note that you'll need a non-linearity between your 'learners' and the 'mixing' layer, eg ReLU or tanh.
You hint at giving each 'learn' its own multinomial draw, or at least, softmax. On the whole, I think it will be more standard to have the mixing layer first, followed by a single softmax and multinomial draw. This will give the least variance, since fewer draws. (for example, you can look at 'variational dropout' paper, https://arxiv.org/abs/1506.02557 , which explicitly merges multiple random draws, to reduce variance, a technique they call 'local reparameterization')
Such a network will look something like:
This then has the following characteristics:
- can include one or more independent learners, each with their own parameters
- can include a prior over the distribution of the output classes
- will learn to mix across the various learners
Note in passing that this is not the only way of combining the learners. We could also combine them in more of a 'highway' type fashion, somewhat like boosting, something like:
In this last network, each learner learns to fix any issues caused by the network so far, rather than creating its own relatively independent prediction. Such an approach can work quite well, ie Boosting, etc.
