1
$\begingroup$

I trained an SVM (RBF kernel, optimized C and G) on a dataset with a balanced class distribution (i.e., 50% positive, 50% negative class samples). Testing the model on a corpus with an unbalanced class distribution (i.e., 1% positive) shows significant overgeneration of the model, as about half of the test instances received a positive class label.

Does this mean that the model takes prior distribution information into account? Can someone help to explain?

$\endgroup$
  • 1
    $\begingroup$ The general answer is "it depends", since the more information your data gives (as compared to prior) the more it affects the results. In many cases large samples make the priors irrelevant, but it is not always the case. See stats.stackexchange.com/questions/200982/… for detailed discussion. $\endgroup$ – Tim Jun 16 '17 at 10:17
2
$\begingroup$

What you are observing is the effect of dataset shift. It occurs when the joint distribution $P(x,y)$ of input ($x$) and label ($y$) vary widely between the training and test datasets. In such cases, the model trained on the training data performs poorly on the test dataset. Prior probability shift is when only the distribution over $y$ changes and everything else stays the same. It is a common issue in simple generative models.

The fundamental assumption of supervised learning is that the joint probability distribution $P(x,y)$ will remain unchanged while training and testing a model. As $P(x,y)=P(y|x)P(x)=P(x|y)P(y)$, the joint distribution over training and test dataset may change if any of the four probability distribution changes.

You can refer to this link which was mentioned in an answer about handling unbalanced dataset.

You can learn more about dataset shift from this book where the term was first used. You can skip to page 12 of the book to know more about prior probability shift.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.