I'm new to Machine Learning and this forum. I have a beginner's doubt regarding imbalanced dataset. Here it goes: I have a binary classification task, where I'm more interested in accurately classifying the positive class (which is in minority in the target population). Unlike the common problem of not having enough positive (minority) class instances in the training set, my training dataset contains the positive class in majority.
Here's the target population composition (which I'd expect to find in the environment where my classifier/model would be deployed):
- Positive Class: ~35%
- Negative Class: ~65%
Here's my Training Set Composition:
- Positive Class: ~95%
- Negative Class: ~5%
As my training set composition drastically differs from the target population composition, will the classification algorithm fail to generalize when classifying instances from the target population? As I mentioned earlier, I'm more interested in accurately classifying the positive class instances, which my training set has in abundance.
I read the following description in a publication on imbalanced datasets: "The purpose of machine learning is for the classifier to estimate the probability distribution of the target population. Since that distribution is unknown we try to estimate the population distribution using a sample distribution. Statistics tells us that as long as the sample is drawn randomly, the sample distribution can be used to estimate the population distribution from where it was drawn. Hence, by learning the sample distribution we can learn to approximate the target distribution."
Since my training dataset can not be considered as a random sample of the target distribution, will this affect the generalization power of my classifier? If so, what shall be done to avoid this? Over/under-sampling? Cost Matrices?
PS: I tried to search previous posts about issues similar to mine, but all of them dealt with the problem of not having sufficient examples for the minority class (which is the exact opposite of my scenario).
Thanks in advance -S