5
$\begingroup$

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/test dataset is different.

But my main concern is which approach among below is correct

Approach 1

Should I pass the entire dataset for cross-validation and get the best model paramters

Approach 2

  1. Do a train test split of data
  2. Pass X_train and y_train for cross-validation (Cross validation will be done only on X_train and y_train. Model will never see X_test, y_test)
  3. Test the model with best parameters obtained from cross-validation of X_train and y_train on X_test and y_test

Concerns with Approach 1

How will I validate the model if it is trained on entire dataset

Concerns with Approach 2

The parameters obtained for this approach will be biased to what data is present in X_train and y_train.How to get rid of this bias

$\endgroup$
1
  • 3
    $\begingroup$ Approach 1 is not appropriate. You need some final measure of out of sample accuracy. It is a better idea to split the data into train and test, and perform CV on the training set, leaving testing to be used only once you have selected a model. WRT your concerns for approach 2, the hope is that the random split used to create the training set is not biased in anyway. We would hope that the data are representative enough so that any sufficiently large subset is also representative. $\endgroup$ Nov 26, 2018 at 6:47

3 Answers 3

1
$\begingroup$

Your approaches are unclear. So, here is my simple explanation of cross validation. Cross-validation is done to tune the hyperparamaters such that the model trained generalizes well (by validating it on validation data). So here is a basic version of held-out cross-validation:

  1. Train test(actually validation) split the data to obtain XTrain, yTrain, XVal, yVal

  2. Select a set of hyperparameter grid you want to search on.

  3. For ith hyperparameter combination:

    a. Train(fit) model on XTrain, yTrain

    b. Evaluate the model

    c. Evaluate the model on XVal, yVal i.e., compute the performance metric (accuracy, auc, f1, etc).

  4. After 3, select the hyperparameter combination which provides best performance metric.

There are other flavors of cross-validation like k-fold cross validation and iterated cross-validation which work better.

EDIT: For doing k-fold cross-validation, you don't need to split the data into training and validation set, it is done by splitting the training data into k-folds, each one of which will be used as a validation set in training the other (k-1) folds together as training set. The evaluation metric will then be the average of the evaluation metrics in the k iterations.

$\endgroup$
5
  • 1
    $\begingroup$ By cross validation I mean I will be using k-fold cross validation $\endgroup$
    – Rookie_123
    Nov 26, 2018 at 9:27
  • 1
    $\begingroup$ My question was while doing k-fold cross validation should I pass the X, y (the entire dataset) or the X_train, y_train and then I will test this model with X_test, y_test $\endgroup$
    – Rookie_123
    Nov 26, 2018 at 9:28
  • $\begingroup$ While doing k-fold cross-validation using say GridSearchCV from sklearn, you should pass the entire training set, but not the test set, since you will be validating on it. I will edit my answer though.. $\endgroup$ Nov 26, 2018 at 10:56
  • 4
    $\begingroup$ Thats means my approach 2 is fairly correct $\endgroup$
    – Rookie_123
    Nov 26, 2018 at 10:59
  • $\begingroup$ And then what about the blogs that say use the average cross validation score to check for overfitting. How does that work? $\endgroup$
    – Dev_Man
    Oct 20, 2019 at 8:32
0
$\begingroup$

As I understand when you do the cross validation in the training dataset you're looking for the best combination of parameters for that model, when you look for the cross validation score in that case you're evaluating the performance of your model in the training dataset (Should be very good) an then you evaluate your model with the test data to see the results but without cross validation. I am not 100% sure about this but at least this is how I understand the process. What I don't understand is how this improve your model avoiding overffiting

$\endgroup$
0
$\begingroup$

Cross-validation involves a trade-off between the amount of data you use for training (i.e., for fitting your model) and the amount of data you use for testing. In order to avoid biasing the analysis we split the data into a training and testing set using simple random sampling without replacement using some stipulated number of data points for each part. Nevertheless, we still have a trade-off --- a data point used for one activity is not used for the other. If you use the entire dataset for training then you have more data for fitting your model, but you have no data left over for testing, so you cannot do cross-validation at all!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.