What is an appropriate strategy for splitting the dataset?
I ask for feedback on the following approach (not on the individual parameters like
n_iter, but if I used
y_test appropriately and if the sequence makes sense):
(extending this example from the scikit-learn documentation)
1. Load the dataset
from sklearn.datasets import load_digits digits = load_digits() X, y = digits.data, digits.target
2. Split into training and test set (e.g., 80/20)
from sklearn.cross_validation import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
3. Choose estimator
from sklearn.svm import SVC estimator = SVC(kernel='linear')
4. Choose cross-validation iterator
from sklearn.cross_validation import ShuffleSplit cv = ShuffleSplit(X_train.shape, n_iter=10, test_size=0.2, random_state=0)
5. Tune the hyperparameters
applying the cross-validation iterator on the training set
from sklearn.grid_search import GridSearchCV import numpy as np gammas = np.logspace(-6, -1, 10) classifier = GridSearchCV(estimator=estimator, cv=cv, param_grid=dict(gamma=gammas)) classifier.fit(X_train, y_train)
6. Debug algorithm with learning curve
X_train is randomly split into a training and a test set 10 times (
n_iter=10). Each point on the training-score curve is the average of 10 scores where the model was trained and evaluated on the first i training examples. Each point on the cross-validation score curve is the average of 10 scores where the model was trained on the first i training examples and evaluated on all examples of the test set.
from sklearn.learning_curve import learning_curve title = 'Learning Curves (SVM, linear kernel, $\gamma=%.6f$)' %classifier.best_estimator_.gamma estimator = SVC(kernel='linear', gamma=classifier.best_estimator_.gamma) plot_learning_curve(estimator, title, X_train, y_train, cv=cv) plt.show()
plot_learning_curve() can be found in the current dev version of scikit-learn (0.15-git).
7. Final evaluation on the test set
7a. Test over-fitting in model selection with nested cross-validation (using the whole dataset)
from sklearn.cross_validation import cross_val_score cross_val_score(classifier, X, y)
Additional question: Does it make sense to replace step 7 by nested cross-validation? Or should nested cv be seen as complementary to step 7
(the code seems to work with k-fold cross validation in scikit-learn, but not with shuffle & split. So
cv needs to be changed above to make the code work)
8. Train final model on whole dataset
EDIT: I now agree with cbeleites that step 7a doesn't make much sense in this sequence. So I wouldn't adopt that.