Starting from Linear Regression, we always divide our dataset into training and test. Once we come to cross validation, this is now training and validation sets. In validation approach, we divide either equally or say into two parts. In Leave one out cross validation (LOOCV) we split the dataset by 1 : n-1, in K-fold we divide the dataset into almost equal k parts.

So, cross validation is just varying the way we divide the datasets?


I'm really not sure I understand your question. To answer what you seem to be asking, no, cross-validation isn't just a matter of dividing a dataset into parts (folds). That's only the first step. The second step is to, for each fold, treat that fold as the test set, training the model on the rest of the dataset. Once the cross-validation procedure is through, every data point has been treated as (a) part of the test set one time, and (b) part of the training set k - 1 times where k is the number of folds. Compare this to the method of setting aside a test set, which causes every data point to be used once in the training set or once in the test set but not both. It's also possible to combine these methods, which is useful if you want to make modeling decisions on the basis of cross-validation before getting a final estimate of test error using the held-out test set.


No, Cross-validation is not just varying the size of training and validation sets. There is a reason for doing so.

It is basically two concepts of Machine Learning,

1) Our algorithm should perform well on "future unseen" data points - that means the data points, which are not in hand (Data which will be affected by future situations), so neither training data nor test data.

e.g Product reviews can change over time, maybe due to reasons like 1) product has been modified in the future or competitor's products are not received in good condition due to changed packaging material, and now the reviews will be little different from what (they used to be) we have in hand, so "unseen future data" may not be "exactly" same as our train or test data. Even in this situation, our algorithm should work well.

2) When we determine K we are using test data and that is somewhat like doing reverse engineering, so we may get 96% accuracy on Test data using k = 6, but it may not work for future unseen data. Here we've already seen(used) the test data while determining K.

If we divide data in cross-validation and on the basis of it we determine K, then we haven't used test data(test data is still unseen) and so the K will be more relevant when it comes to future unseen data.

K-fold cross-validation we divide data normally in 3 parts - 60% training data, 20% cross-validation data, 20% training data.

Problem : Of the total n data points, we are using only 60% of data to compute nearest neighbors. Is there a way to use 80% of the data (train 60% + cross-validation 20%)? As most of the time, more the training data-> better is the algorithm. K-fold cross-validation is a solution here.

Randomly break train data into 4 parts: D1=20% of data, D2=20% of data, D3= 20% of data, D4 = 20% of data.

Now for different values of K, use combinations of 3 parts as training data and the one which is left out is a cross-validation data. e.g

 **1)** D1,D2,D3 <- Train & D4 <- CV

 **2)** D1,D2,D4 <- Train & D3 <- CV

 **3)** D1,D3,D4 <- train & D2 <- CV 

 **4)**D2,D3,D4 <- Train & D1 <- CV 

This way we can use every part of training data for cross-validation and for training, so we are not wasting data.

Limitation: - Time it takes to compute the optimal/best K in KNN increases by K times if we use K-fold cross-validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.