I am constructing a model for the prediction of a binary (Yes/No) outcome. I have a learning sample that gives the machine 1500 examples of the "Yes" group and 500 example of the "No" group. Should I be using all the data I have for input to learn the machine? Would this be biased towards the "Yes"?
I had the thought of giving 500 "Yes" and 500 "No" examples, but I am not sure if this is going to positively or negatively my future predictions.
Thanks.
Most learning algorithms have a way to deal with skewed data sets. In general, use as much as you can for learning to increase generalization performance.
Questions you should ask yourself are "Why is my training set biased?" and "What will the data look like at application time?" If the bias is a natural property of the data, that bias should probably be represented in the training set as well. If the bias is a selection bias due to the way the data was collected, you should try to compensate.
For most classifiers, training algorithms and datasets, the bias will affect the prediction! However, this bias might be desirable. When doing face recognition, for example, and your classifier is going to be applied to your photo albums, you want your training algorithm to be able to use the fact that most pictures will be pictures of you. However, if your classifier is going to be used with photo albums of all kinds of people, and the only reason your training set contains more pictures of you is because they were easier to get, you should try to compensate.
You don't necessarily need to throw away data to compensate for the bias. Many training algorithms allow you to weight your training examples (gradient descent and expectation maximization, for example). To get rid of any bias, try to weight positive examples by $1/\#Yes$ and negative examples by $1/\#No$, where $\#Yes$ and $\#No$ are the number of positive and negative examples, respectively.
Say your dataset is $\mathcal{D} = \{ (x_n, y_n) : n=1,\dots,N\} \subseteq \mathbb{R}^N \times \{0, 1\}$. Further let $\mathcal{D}_1 = \{ (x_n, y_n) : y_n = 1 \} \subseteq \mathcal{D}$ be your positive examples, and $\mathcal{D}_0 \subseteq \mathcal{D}$ be your negative examples. Typically, the objective function optimized by your classifier during training can be written
$$F(\theta) = \sum_{(x_n, y_n) \in \mathcal{D}} f(x_n, y_n; \theta),$$
where $\theta$ are some parameters that are optimized. We can split that objective function into a sum over positive and negative examples,
$$F(\theta) = \sum_{(x_n, y_n) \in \mathcal{D}_0} f(x_n, y_n; \theta) + \sum_{(x_n, y_n) \in \mathcal{D}_1} f(x_n, y_n; \theta).$$
If the number of positive examples $\#\mathcal{D}_1$ is much larger than the number of negative examples $\#\mathcal{D}_0$, then the second term will dominate and the optimizer will try to make sure that positive examples are correctly classified and will care less about negative examples being wrongly classified. To compensate, we could weight the individual examples or the two terms,
$$\frac{1}{\#\mathcal{D}_0} \sum_{(x_n, y_n) \in \mathcal{D}_0} f(x_n, y_n; \theta) + \frac{1}{\#\mathcal{D}_1} \sum_{(x_n, y_n) \in \mathcal{D}_1} f(x_n, y_n; \theta).$$
Now both terms contribute equally to the overall classification error or whatever you are trying to optimize.
