4
$\begingroup$

As recommended by Andrew Ng in his great course on machine learning, I would like to plot the learning curves for experiments I am running with Random Forest and SVM algorithms.

The learning curves are computed as the cost minimized during the training vs the number of samples for the training and the testing sets and allow to detect high variance or high bias problems.

I'm using scikit-learn and I'm aware of sklearn.learning_curve.learning_curve, but it computes the classification scores for different training set sizes and I'm wondering whether it is the same as using the cost.

Is using the classification score the correct way to plot the learning curve for a classification process in order to diagnose high variance or bias? Or is there any cost I could use?

$\endgroup$

2 Answers 2

2
$\begingroup$

In fact, you can define your own error function and pass it to the validation_curve() function as so:

def rms_error(model, X, y):
    y_pred = model.predict(X)
    return np.sqrt(np.mean((y - y_pred) ** 2))

val_train, val_test = validation_curve(PolynomialRegression(), X, y,
                                       'polynomialfeatures__degree',
                                       degree, cv=7, scoring=rms_error)
$\endgroup$
1
$\begingroup$

As far as i remember Andreg Ng's course (i watched it and its the best resource for learning about learning curves imho), you want to plot two curves:

  1. Train error - The trained model applied on the training data itself.

  2. Validation error - Crossvalidation performed on the training set.

Both error curves are plotted against an increasing number of samples (= "what i referred to as training set") on the x-axis. Depending on the shapes of the curves you can draw conclusion about bias / variance.

I am not an expert with scikit learn, but learning_curve looks like it can only return the crossvalidation error. Nevertheless computing the train error is even simpler, because all you have to do is to generate a prediction on all items of the train set and evaluate it (using the same metric which whas used for the cv). However the training subsets on which you compute the train / validation error should be the same at every step size.

$\endgroup$
3
  • $\begingroup$ The learning_curve method splits the data and returns the classification score vs the data set size for both the training set and the test set. My question is about whether using the classification scores is the same as using the model fitting error. $\endgroup$
    – jul
    Apr 12, 2015 at 16:59
  • $\begingroup$ The model fitting error depends on the classification metrics. For evaluation you usually choose one or more metrices (depending on your classification problem). Lets assume the easiest case, you want to perform binary classification. Then you would choose the accuracy metric (scoring = 'accuracy') and you get the model fitting error by $err = 1 - acc$. So yes it is the same, as long as you refer to the same metric (= scoring function) in validation and cv. You can create learning curves with many metrices (fscore, accuracy very common) and the learning curve plots the respective error. $\endgroup$ Apr 12, 2015 at 17:34
  • $\begingroup$ Maybe my question was not clear enough. What I mean by "model fitting error" is basically the cost which is minimized during the training process. I'll edit my question. I know the classification score is correlated to that, but it is not the same. $\endgroup$
    – jul
    Apr 13, 2015 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.