I have doubts regarding cross validation and train / test split. I understand the following... please correct me if I am wrong.

If I use train and test split, with the following code, I will be able to:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_train)
metrics.accuracy_score(y_train, y_pred)

y_pred = clf.predict(X_test)
metrics.accuracy_score(y_test, y_pred)

1 - Get a model

2 - Obtain a score of the model on the same data with which he was trained

3 - Get a score of the model, testing it with data that he has never seen

Now my doubt with cross validation is ... it is useful to use cross validation after the split, and apply this cross validation only on the train data?

I understand that I do not get any model

If what I get are only scores, so that they are useful to me?   Do they best represent the train score, or does it represent the test score better?

scores = cross_val_score(clf, X_train, y_train, cv=10)

Thank you very much for the help

  • $\begingroup$ You should be using cross-validation solely on the training data (i.e. after the split) to tune your model (e.g. to choose the hyperparameters or even to choose the type of model). It may give you an estimate for the out-of-sample error, but in some cases not a particularly good one if there is overfitting $\endgroup$ – Henry Jun 24 '17 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.