Plotting learning curves for any classification algorithm As recommended by Andrew Ng in his great course on machine learning, I would like to plot the learning curves for experiments I am running with Random Forest and SVM algorithms.
The learning curves are computed as the cost minimized during the training vs the number of samples for the training and the testing sets and allow to detect high variance or high bias problems.
I'm using scikit-learn and I'm aware of sklearn.learning_curve.learning_curve, but it computes the classification scores for different training set sizes and I'm wondering whether it is the same as using the cost.
Is using the classification score the correct way to plot the learning curve for a classification process in order to diagnose high variance or bias? Or is there any cost I could use?
 A: In fact, you can define your own error function and pass it to the validation_curve() function as so:
def rms_error(model, X, y):
    y_pred = model.predict(X)
    return np.sqrt(np.mean((y - y_pred) ** 2))

val_train, val_test = validation_curve(PolynomialRegression(), X, y,
                                       'polynomialfeatures__degree',
                                       degree, cv=7, scoring=rms_error)

A: As far as i remember Andreg Ng's course (i watched it and its the best resource for learning about learning curves imho), you want to plot two curves:


*

*Train error - The trained model applied on the training data itself. 

*Validation error - Crossvalidation performed on the training set. 
Both error curves are plotted against an increasing number of samples (= "what i referred to as training set") on the x-axis.
Depending on the shapes of the curves you can draw conclusion about bias / variance.
I am not an expert with scikit learn, but learning_curve looks like it can only return the crossvalidation error. Nevertheless computing the train error is even simpler, because all you have to do is to generate a prediction on all items of the train set and evaluate it (using the same metric which whas used for the cv). 
However the training subsets on which you compute the train / validation error should be the same at every step size.
