My problem is this: I have an ensemble of predictors that each produces a distribution over a set of classes.

What I would like to do is to first have a non-informative prior about how this label distribution looks like, and then update that prior with the prediction of each member of the ensemble.

So I thought of using a non-informative Dirichlet prior, which I then update with each sample distribution that comes in as a prediction.

My question is: Is this approach valid, and if yes how would I update my prior so that, it becomes more defined as more samples accumulate?


Dirichlet prior is an appropriate prior, and is the conjugate prior to a multinomial distribution. However, it seems a bit tricky to apply this to the output of a multinomial logistic regression, since such a regression has a softmax as the output, not a multinomial distribution. However, what we can do is sample from a multinomial, whose probabilities are given by the softmax.

If we draw this as a neural network model, it looks like:

enter image description here

We can easily sample from this, in the forwards direction. How to handle backwards direction? We can use the reparameterization trick, from Kingma's 'Auto-encoding variational Bayes' paper, https://arxiv.org/abs/1312.6114 , in other words, we model the multinomial draw as a deterministic mapping, given the input probability distribution, and a draw from a standard Gaussian random variable:

$$ x_{\text{out}} = g(\mathbf{\mathbf{x}_{\text{in}}}, \mathbf{\epsilon}) $$

where: $\mathbf{\epsilon} \sim \mathcal{N}(0, 1)$

So, our network becomes:

enter image description here

So, we can forward propagate mini-batches of data examples, draws from the standard normal distribution, and back-propagate through the network. This is fairly standard and widely used, eg the Kingma VAE paper above.

A slight nuance is, we are drawing discrete values from a multinomial distribution, but the VAE paper only handles the case of continuous real outputs. However, there is a recent paper, the Gumbel trick, https://casmls.github.io/general/2017/02/01/GumbelSoftmax.html , ie https://arxiv.org/pdf/1611.01144v1.pdf ,and https://arxiv.org/abs/1611.00712 , which allows draws from discrete multinomial papers.

The Gumbel trick formulae gives the following output distribution:

$$ p_{\alpha, \lambda}(x) = (n-1)! \lambda^{n-1} \prod_{k=1}^n \left( \frac{\alpha_k x_k^{-\lambda-1}} {\sum_{i=1}^n \alpha_i x_i^{-\lambda}} \right) $$

The $\alpha_k$ here are prior probabilities for the various categories, which you can tweak, to push your initial distribution towards how you think the distribution could be distributed initially.

Thus we have a model that:

  • contains a multinomial logistic regression (the linear layer followed by the softmax)
  • adds a multinomial sampling step at the end
  • which includes a prior distribution over the probabilities
  • can be trained, using Stochastic Gradient Descent, or similar


So, the question asks:

"is it possible to apply this kind of technique when we have multiple predictions (and each prediction can be a softmax, like above) for a single sample (from an ensemble of learners)." (see comments below)

So: yes :). It is. Using something like multi-task learning, eg http://www.cs.cornell.edu/~caruana/mlj97.pdf and https://en.wikipedia.org/wiki/Multi-task_learning . Except multi-task learning has a single network, and multiple heads. We will have multiple networks, and a single head.

The 'head' comprises an extract layer, which handles 'mixing' between the nets. Note that you'll need a non-linearity between your 'learners' and the 'mixing' layer, eg ReLU or tanh.

You hint at giving each 'learn' its own multinomial draw, or at least, softmax. On the whole, I think it will be more standard to have the mixing layer first, followed by a single softmax and multinomial draw. This will give the least variance, since fewer draws. (for example, you can look at 'variational dropout' paper, https://arxiv.org/abs/1506.02557 , which explicitly merges multiple random draws, to reduce variance, a technique they call 'local reparameterization')

Such a network will look something like:

enter image description here

This then has the following characteristics:

  • can include one or more independent learners, each with their own parameters
  • can include a prior over the distribution of the output classes
  • will learn to mix across the various learners

Note in passing that this is not the only way of combining the learners. We could also combine them in more of a 'highway' type fashion, somewhat like boosting, something like:

enter image description here

In this last network, each learner learns to fix any issues caused by the network so far, rather than creating its own relatively independent prediction. Such an approach can work quite well, ie Boosting, etc.

  • 1
    $\begingroup$ Thanks this is really useful, although the question still has an unanswered aspect: Is it possible to apply this kind of technique when we have multiple predictions (and each prediction can be a softmax, like above) for a single sample (from an ensemble of learners) . My guess would be not, because the samples are not i.i.d. $\endgroup$ – Bar Oct 10 '17 at 10:31
  • $\begingroup$ @Bar : Good point. Updated for multiple learners $\endgroup$ – Hugh Perkins Oct 10 '17 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.