I am quite confused about the objective function of the bayesian model averaging in the paper "Bayesian Averaging of Classifiers and the overfitting Problem".1

On the section 2, here is the first equation:

Let $n$ be the training set size, $\mathbf{x}$ examples in the training set, $\mathbf{c}$ the corresponding class labels and $h$ a model (or hypothesis) in the model space $H$. Then, by Bayes’ theorem, and assuming the examples are drawn independently, the posterior probability of $h$ given $(\mathbf{x}, \mathbf{c})$ is given by:

$Pr(h|\mathbf{x}, \mathbf{c})=\frac{Pr(h)}{Pr(\mathbf{x}, \mathbf{c})}\prod_{i=1}^{n}Pr(\mathbf{x_i},\mathbf{c_i}|h)$ (1)

where $Pr(h)$ is the prior probability of $h$, and the product of $Pr(\mathbf{x_i},\mathbf{c_i}|h)$ terms is the likelihood.

I could understand that the Eq(1) uses conditional independence.

In order to compute the likelihood it is necessary to compute the probability of a class label $\mathbf{c_i}$ given an unlabeled example $\mathbf{x_i}$ and a hypothesis $h$, since $Pr(\mathbf{x_i}, \mathbf{c_i}|h) = Pr(\mathbf{x_i}|h)Pr(\mathbf{c_i}|\mathbf{x_i}, h)$. This probability, $Pr(\mathbf{c_i}|\mathbf{x_i}, h)$, can be called the noise model, and is distinct from the classification model $h$, which simply produces a class prediction with no probabilities attached.

I can understand the above as well.


Finally, an unseen example $x$ is assigned to the class that maximizes: $Pr(c|x,\mathbf{x},\mathbf{c}, H)=\sum_{h\in H}Pr(c|x,h)Pr(h|\mathbf{x},\mathbf{c})$ (4)

I have two questions:

  1. I don't understand how to deduce the Eq(4);
  2. In those euqations of Bayesian Model Averaging, which are variables? I don't understand how to train it.

Thank you in advance.

1 Domingos, P., (2000) "Bayesian Averaging of Classifiers and the Overfitting Problem" Proceedings of the Seventeenth International Conference on Machine Learning, pp.223-230


1 Answer 1


After days' consideration, I think I might know how to deduce:

$Pr(c|x,\mathbf{x}, \mathbf{c}, H) =\frac{\sum_{h\in H} Pr(\mathbf{x}, \mathbf{c}) Pr(x) Pr(h|\mathbf{x}, \mathbf{c}) Pr(c|x, h)}{\sum_{h\in H} Pr(\mathbf{x}, \mathbf{c}) Pr(x) Pr(h|\mathbf{x}, \mathbf{c})} =\frac{\sum_{h\in H}Pr(h|\mathbf{x}, \mathbf{c}) Pr(c|x, h)}{\sum_{h\in H} Pr(h|\mathbf{x}, \mathbf{c})} =\sum_{h\in H}Pr(h|\mathbf{x}, \mathbf{c}) Pr(c|x, h)$

I can't be certain though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.