0
$\begingroup$

I am trying to compare numerous neural network models in terms of log likelihood, for example there are two given here...

MC Dropout - https://arxiv.org/abs/1506.02142 Deep Ensembles - http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles.pdf

In the first example, they use the logsumexp trick in order to calculate the log probability over the number of samples $K$ they take (https://github.com/yaringal/DropoutUncertaintyExps)...

$$ \log p(y_i | x_i) = \log \frac{1}{K} \sum_k \exp{\log{p(y_{ik} | x_{ik})}} $$

Which measures the average log probability over each sample. This gives a final log probability but we do not have the parameters a density to go along with this distribution (as far as I know). So if I wanted to graph out the mean and standard deviation of this, I could not do it.

On the other hand, the second paper I posted calculates log probability by taking a mixture of Gaussians for the predictions from different models (similar to sample $K$ above) and then calculates the log probability based on this resulting density.

$$ \mu_* = \frac{1}{K} \sum_k \mu_k \\ \sigma^{2}_* = \frac{1}{K} \sum_k(\sigma^2_k + \mu^2_k) - \mu_*^2 \\ \log p(y_i | x_i) = \log \mathcal{N}(y_i| \mu_*, \sigma^2_*) $$

Which gives a different numerical result than the first method.

In my opinion, I think the second method is better because in a real test situation, we need to have a final density in order to evaluate the prediction of the model, right?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.