0
$\begingroup$

Suppose I have some classifier which is reasonably good at discriminating classes. I have a new dataset which I know has a very unbalanced class distribution, but I don’t know anything about this distribution, so I use a uniform prior distribution for my classifier. Because my classifier is a good discriminator, the output posterior distribution is much closer to the actual distribution of the dataset than the uniform distribution that I originally assumed.

My understanding is that in the next experiment with similar parameters, I can use this posterior distribution as the prior to improve the classification output. My question is: can I also use this posterior distribution with the original dataset to improve the output?

I’ve actually done this using a neural network as a classifier with thresholding according to Bayes’ theorem (essentially, prediction thresholds are proportional to the given prior distribution). Running prediction with uniform priors to get a posterior probability distribution, and then re-running prediction with that new distribution as the priors yields much better results compared to using just the uniform distribution. In fact, doing this over and over again recursively until predictions converge yields the best results, and comes extremely close to predicting the actual class distribution of the dataset.

I hope this question makes sense — I’m fairly new to probability theory so I’m mainly concerned with whether or not this approach is valid, or if my results are somehow spurious.

$\endgroup$
1
$\begingroup$

The procedure, that you described, has a similar effect to artificially inflating the sample size. It's akin to cloning the observations of the sample. Say, you take each observation and create 9 more copies of it and run the Bayesian update on the 10 times bigger sample.

Here's an example from the Wiki article.

Suppose there is a school having 60% boys and 40% girls as students. The girls wear trousers or skirts in equal numbers; all boys wear trousers. An observer sees a (random) student from a distance; all the observer can see is that this student is wearing trousers. What is the probability this student is a girl?

You get the posterior $p(G|T)=1/4$ of a girl from observing a student in trousers. If you run Bayes theorem once more you get $P^{(2)}(G|T)=1/7$

What do you think will happen if you keep running Bayes over and over plugging the posterior as prior? Basically, you'll make it look like it's certainly a boy: $P(G|T)\to 0$. This is effectively saying that girls don't wear trousers based on just one observation of a student in trousers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.