1
$\begingroup$

I am constructing a model for the prediction of a binary (Yes/No) outcome. I have a learning sample that gives the machine 1500 examples of the "Yes" group and 500 example of the "No" group. Should I be using all the data I have for input to learn the machine? Would this be biased towards the "Yes"?

I had the thought of giving 500 "Yes" and 500 "No" examples, but I am not sure if this is going to positively or negatively my future predictions.

Thanks.

$\endgroup$
2
$\begingroup$

Most learning algorithms have a way to deal with skewed data sets. In general, use as much as you can for learning to increase generalization performance.

$\endgroup$
  • $\begingroup$ Thanks for your reply. Does apply even if the data was....immensely skewed? like 2500 Yes to 400 No? $\endgroup$ – Error404 Aug 9 '13 at 9:26
  • 1
    $\begingroup$ Yes, it does. Many algorithms can deal with such things properly. $\endgroup$ – Marc Claesen Aug 9 '13 at 9:27
  • 2
    $\begingroup$ @Error404 That's skewed, but not what I'd called immensely skewed. Many applications involve probabilities of $10^{-3}$ or less: rare diseases, insurance risk, credit risk, etc. $\endgroup$ – Hong Ooi Aug 9 '13 at 9:30
2
$\begingroup$

Questions you should ask yourself are "Why is my training set biased?" and "What will the data look like at application time?" If the bias is a natural property of the data, that bias should probably be represented in the training set as well. If the bias is a selection bias due to the way the data was collected, you should try to compensate.

For most classifiers, training algorithms and datasets, the bias will affect the prediction! However, this bias might be desirable. When doing face recognition, for example, and your classifier is going to be applied to your photo albums, you want your training algorithm to be able to use the fact that most pictures will be pictures of you. However, if your classifier is going to be used with photo albums of all kinds of people, and the only reason your training set contains more pictures of you is because they were easier to get, you should try to compensate.

You don't necessarily need to throw away data to compensate for the bias. Many training algorithms allow you to weight your training examples (gradient descent and expectation maximization, for example). To get rid of any bias, try to weight positive examples by $1/\#Yes$ and negative examples by $1/\#No$, where $\#Yes$ and $\#No$ are the number of positive and negative examples, respectively.

Say your dataset is $\mathcal{D} = \{ (x_n, y_n) : n=1,\dots,N\} \subseteq \mathbb{R}^N \times \{0, 1\}$. Further let $\mathcal{D}_1 = \{ (x_n, y_n) : y_n = 1 \} \subseteq \mathcal{D}$ be your positive examples, and $\mathcal{D}_0 \subseteq \mathcal{D}$ be your negative examples. Typically, the objective function optimized by your classifier during training can be written

$$F(\theta) = \sum_{(x_n, y_n) \in \mathcal{D}} f(x_n, y_n; \theta),$$

where $\theta$ are some parameters that are optimized. We can split that objective function into a sum over positive and negative examples,

$$F(\theta) = \sum_{(x_n, y_n) \in \mathcal{D}_0} f(x_n, y_n; \theta) + \sum_{(x_n, y_n) \in \mathcal{D}_1} f(x_n, y_n; \theta).$$

If the number of positive examples $\#\mathcal{D}_1$ is much larger than the number of negative examples $\#\mathcal{D}_0$, then the second term will dominate and the optimizer will try to make sure that positive examples are correctly classified and will care less about negative examples being wrongly classified. To compensate, we could weight the individual examples or the two terms,

$$\frac{1}{\#\mathcal{D}_0} \sum_{(x_n, y_n) \in \mathcal{D}_0} f(x_n, y_n; \theta) + \frac{1}{\#\mathcal{D}_1} \sum_{(x_n, y_n) \in \mathcal{D}_1} f(x_n, y_n; \theta).$$

Now both terms contribute equally to the overall classification error or whatever you are trying to optimize.

$\endgroup$
  • $\begingroup$ Thanks for the detailed explanation, I am still wondering about a couple of things though. 1- I did not get how to get rid of the bias by dividing 1/#Yes and 1/#No, what do I do after I get these values? 2- The answer I had by Marc (you can see it above) was that the algorithms "deals" with skewed data rather than "applies" the skewing to the future samples. To be honest, I would think of the case the way you explained it, but does that make the above answer inaccurate? $\endgroup$ – Error404 Aug 9 '13 at 11:47
  • 1
    $\begingroup$ You don't divide the data, you weight the data. I added a little bit more formal explanation to make this more clear. I can't say what kind of algorithms or automatic mechanisms @MarcClaesen had in mind when he wrote his answer. $\endgroup$ – Lucas Aug 9 '13 at 12:29
  • $\begingroup$ I will check that. Cheers mate $\endgroup$ – Error404 Aug 9 '13 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.