12
$\begingroup$

I created this learning curve and I want to know if my SVM model suffers from bias or variance? How can I conclude that from this graph?

learning cuve

$\endgroup$
13
$\begingroup$

Part 1: How to read learning curve

Firstly, we should focus on the right side of the plot, where there are sufficient data for evaluation.

  • If two curves are "close to each other" and both of them but have a low score. The model suffer from an under fitting problem (High Bias)

  • If training curve has a much better score but testing curve has a lower score, i.e., there are large gaps between two curves. Then the model suffer from an over fitting problem (High Variance)

Part 2: My assessment for the plot you provided

From the plot it is hard to say if the model is good or not. It is possible that you have a really "easy problem", a good model can achieve 90%. On the other hand, it is possible you have a really "hard problem" that the best thing we can do is achieving 70%. (Note that, you may not expect you will have a perfect model, say score is 1. How much you can achieve depends on how much noise in your data. Suppose your data has a lot of data points have EXACT feature but different labels, no matter what you do, you cannot achieve 1 on score.)

Another problem in your example is that 350 examples seems to be too small in a real world application.

Part 3: More suggestions

To get a better understanding, you can do following experiments to experience under fitting an over fitting and observe what will happen in learning curve.

  • Select a very complicated data say MNIST data, and fit with a simple model, say linear model with one feature.

  • Select a simple data, say iris data, fit with a complexity model, say, SVM.


Part 4: Other examples

In addition, I will give two examples related to under fitting and over fitting. Note this is not learning curve, but the performance respect to number of iterations in gradient boosting model, where more iterations will have more chance of over fitting. The x axis shows number of iterations, and y axis shows the performance, which is negative Area Under ROC (the lower the better.)

The left subplot does not suffer from over fitting (well also not under fitting since the performance is reasonably good) but right one suffers from over fitting when number of iterations is large.

enter image description here

$\endgroup$
  • $\begingroup$ Thank you hxd1011! Do you agree with me if I say that my model suffers from a little variance and a little bias (since score is not 1)? $\endgroup$ – Afke Jun 27 '16 at 13:34
  • $\begingroup$ @Papie I think you model is fine ... 80% on score is not bad and two curves are close. The only problem is examples are up to 350 which might be too small in a real world application. $\endgroup$ – Haitao Du Jun 27 '16 at 13:37
  • $\begingroup$ @Papie Also, you may not expect you will have a perfect model, say score is 1. How much you can acheive depends on how much noise in your data. Suppose your data has a lot of data points have EXACT feature but different labels, no matter what you do, you cannot achieve 1 on score. $\endgroup$ – Haitao Du Jun 27 '16 at 13:41
  • 11
    $\begingroup$ I think his "80% on score is not bad" is not a good way to think about things. There's no global score that is good, it depends heavily on the problem that is being solved, more specifically, on the signal to noise ratio in the process and the data that is available. You point this out in your answer, so I would remove the "not bad" comment. $\endgroup$ – Matthew Drury Jun 27 '16 at 16:53
  • 1
    $\begingroup$ @MatthewDrury Thanks for suggestions, answer revised! $\endgroup$ – Haitao Du Jun 27 '16 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.