3
$\begingroup$

In a supervised learning problem, I have a training dataset $D$ comprised of samples $x$ and their corresponding labels $\omega$. From this data, I attempt to learn the true distributions $p(x|\omega)$ with some estimate $\hat{p}(x|\omega)$ . This estimate could be a simple parametric distribution ($p(x|\theta_\omega)$ where I learned $\theta$ from training data for each class) or it can be the full bayesian distribution ($p(x|D_\omega)$ with the parameter integrated out).

To classify new instances, I want to use the Bayes classifier that maximizes the posterior given the new instances and the training data above some confidence value. So for example if I have 3 independent samples, I can say $\hat{p}(\omega|x_1,x_2,x_3) \propto \hat{p}(x_1 | \omega) \hat{p}(x_2 | \omega) \hat{p}(x_3 | \omega) \hat{p}(\omega)$.

My question is: how do I deal with data that is highly unlikely (small $\hat{p}(x|\omega)$) under all classes? What is the bayesian approach to do this? I can think of heuristics like if the likelihoods for a given sample $x$ fall below a threshold, I reject it and try to collect new evidence. The threshold could be computed by finding the regions where most of the mass is located and samples outside of that are outliers. If I had the true distributions, then I wouldn't have this problem, but since I have estimates, I want to be able to consider the case that my model may be wrong. Rejecting a sample doesn't sound very Bayesian to me. To illustrate what I mean, I can provide the following example:

I have 2 classes with equal priors $p(\omega)$ and the 2 estimated conditional distributions: $N(0,0.1)$ for $\omega_1$ and $N(0.1,0.1)$ for $\omega_2$. If I want to classify new instances and I measure $x_1=0.043$, I get $\hat{p}(\omega_1 | x_1) = 0.5175$ and $\hat{p}(\omega_2 | x_1) = 1-0.5175$. Since I'm not confident about the class, I measure a new sample $x_2 = 0.042$. Then, $\hat{p}(\omega_1 | x_1,x_2) = 0.5374$ and $\hat{p}(\omega_2 | x_1,x_2) = 1-0.5374$. It looks that I'm leaning towards class 1. However, if I measure a new sample $x_3 = 3$, I get that $\hat{p}(\omega_2 | x_1,x_2,x_3) = 1$ due to the fact that the likelihood $\hat{p}(x_3 | \omega_2) >> \hat{p}(x_3 | \omega_1)$. Since I only have estimates of the true distributions, $x_3$ seems like an outlier and shouldn't have that much weight in classification. What is the Bayesian way of dealing with this problem?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A solution could consists in using heavy tail distributions such as a Cauchy to model your $p(x|w_i)$ (instead of gaussian in your example) so that large deviation of a few observation does not over-influence the classification.

Another solution could be to be use a mixture of your learned distribution and of another large standard deviation-distribution to achieve a similar purpose.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Ok, so have a model like $\hat{p}_{new}(x|\omega) = (1-\epsilon) \hat{p}(x|\omega) + \epsilon p_{outlier}(x)$ ? where $p_o(x)$ is the broad outlier distribution. $\endgroup$
    – Damian
    Commented Apr 27, 2016 at 18:21
  • $\begingroup$ @Damian Yes and you could learn $\epsilon$ as well as $p_{outlier}$ $\endgroup$
    – beuhbbb
    Commented Apr 27, 2016 at 18:33
  • $\begingroup$ The mixture solution for handling outliers goes back to Box & Tiao (1968). $\endgroup$
    – Durden
    Commented Mar 12 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.