In a supervised learning problem, I have a training dataset $D$ comprised of samples $x$ and their corresponding labels $\omega$. From this data, I attempt to learn the true distributions $p(x|\omega)$ with some estimate $\hat{p}(x|\omega)$ . This estimate could be a simple parametric distribution ($p(x|\theta_\omega)$ where I learned $\theta$ from training data for each class) or it can be the full bayesian distribution ($p(x|D_\omega)$ with the parameter integrated out).

To classify new instances, I want to use the Bayes classifier that maximizes the posterior given the new instances and the training data above some confidence value. So for example if I have 3 independent samples, I can say $\hat{p}(\omega|x_1,x_2,x_3) \propto \hat{p}(x_1 | \omega) \hat{p}(x_2 | \omega) \hat{p}(x_3 | \omega) \hat{p}(\omega)$.

My question is: how do I deal with data that is highly unlikely (small $\hat{p}(x|\omega)$) under all classes? What is the bayesian approach to do this? I can think of heuristics like if the likelihoods for a given sample $x$ fall below a threshold, I reject it and try to collect new evidence. The threshold could be computed by finding the regions where most of the mass is located and samples outside of that are outliers. If I had the true distributions, then I wouldn't have this problem, but since I have estimates, I want to be able to consider the case that my model may be wrong. Rejecting a sample doesn't sound very Bayesian to me. To illustrate what I mean, I can provide the following example:

I have 2 classes with equal priors $p(\omega)$ and the 2 estimated conditional distributions: $N(0,0.1)$ for $\omega_1$ and $N(0.1,0.1)$ for $\omega_2$. If I want to classify new instances and I measure $x_1=0.043$, I get $\hat{p}(\omega_1 | x_1) = 0.5175$ and $\hat{p}(\omega_2 | x_1) = 1-0.5175$. Since I'm not confident about the class, I measure a new sample $x_2 = 0.042$. Then, $\hat{p}(\omega_1 | x_1,x_2) = 0.5374$ and $\hat{p}(\omega_2 | x_1,x_2) = 1-0.5374$. It looks that I'm leaning towards class 1. However, if I measure a new sample $x_3 = 3$, I get that $\hat{p}(\omega_2 | x_1,x_2,x_3) = 1$ due to the fact that the likelihood $\hat{p}(x_3 | \omega_2) >> \hat{p}(x_3 | \omega_1)$. Since I only have estimates of the true distributions, $x_3$ seems like an outlier and shouldn't have that much weight in classification. What is the Bayesian way of dealing with this problem?


A solution could consists in using heavy tail distributions such as a Cauchy to model your $p(x|w_i)$ (instead of gaussian in your example) so that large deviation of a few observation does not over-influence the classification.

Another solution could be to be use a mixture of your learned distribution and of another large standard deviation-distribution to achieve a similar purpose.

  • $\begingroup$ Ok, so have a model like $\hat{p}_{new}(x|\omega) = (1-\epsilon) \hat{p}(x|\omega) + \epsilon p_{outlier}(x)$ ? where $p_o(x)$ is the broad outlier distribution. $\endgroup$ – Damian Apr 27 '16 at 18:21
  • $\begingroup$ @Damian Yes and you could learn $\epsilon$ as well as $p_{outlier}$ $\endgroup$ – peuhp Apr 27 '16 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.