I have a task of Relationship extraction. There are some set of predefined relations in the corpus. I need to train classifier to recognize the type of relation or the lack of relation between every pair of nouns.

I chose to use a multi-class implementation of SVM. The problem was that I have different number of examples for different relations, for example, for relation type 1 I have 1000 examples in gold standard annotation, however for relation type 2 only 50 examples. The result of classification was very good result on type 1 and very bad on type 2 on validation set.

The solution was to use the same examples from the type 2 several times in my train set, such that the overall number of example of type 1 is close to the number of examples of type 2.

It gives pretty good result on test set. The question is how to explain this, why it actually works. For me it looks counter-intuitive, I didn't expect that using the same examples few times could help.


What you've done is just oversampling. This technique is common when working with unbalanced datasets, see this answer I gave on the topic.

To understand why it helps one needs to think about the objective one is optimizing. Typically the objective will be to minimize a sum over instance losses, possibly with constraints. Something like

$$ J(\theta) = \sum_i \ell(y_i, f(x_i)). $$

By adding multiple copies of the same instance to your dataset you're essentially telling your optimization procedure you care more about getting that instance wrong than others. Rather than actually adding $k$ copies of an instance $x_i$ to you dataset you could achieve a similar result by multiplying the instance loss for example $x_i$ by $k$.

  • $\begingroup$ I am try to understand how it's correlates with assumption that test set has the same distribution as train set, so we can have relatively good model if train and test has the same distribution. But by over sampling I change the distribution, therefore the performance must be changed too. It's more philosophical question, but without doing over sampling I am not able successful recognize the target concept, which has a small probability. $\endgroup$ – user16168 Jan 17 '14 at 8:15
  • $\begingroup$ I think you're conflating two things, good as in low error and good as in correctly identifies the concept. If $P(Y=2) << P(Y=1)$ then a good then a good (low error) hypothesis may be the classifier $f(x) = 1$ for all $x$. However, if you care about about correctly identifying class $2$, this is not so good. $\endgroup$ – alto Jan 17 '14 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.