Difference between Cross validation,GridSearchCV and does cross validation refer the train test split? what is difference between Cross Validation and GridSearchCV and i know how train test split works but i heard many times cross validation, does it refer the same (train test split)?
Can someone please give clear idea with example:-
Thanks in advance.
 A: Cross Validation(CV) or K-Fold Cross Validation (K-Fold CV) is very similar to what you already know as train-test split. When people refer to cross validation they generally mean k-fold cross validation. In k-fold cross validation what you do is just that you have multiple(k) train-test sets instead of 1. This basically means that in a k-fold CV you will be training your model k-times and also testing it k-times. The purpose of doing this is that in a single train-test split, the test part of your data that you chose might be really easy to predict and your model will perform extremely well on it but not exactly so for your actual test sets which ultimately will not be a good model. Hence, you need to use a k-fold CV method. For example, in a 4 fold cross-validation, you will divide your training data into 4 equal parts. In the first step, you keep one part out of the 4 as the set you will test upon and train on the remaining 3. This one part you left out is called the validation set and the remaining 3 becomes your training set. You keep repeating this 4 times but you will be using a different part out of the 4 each time to test your model upon. K-fold cross validation can essentially help you combat overfitting too. There are different ways to do k-fold cross validation like stratified-k fold cv, time based k-fold cv, grouped k-fold cv etc which will depend on the nature of your data and the purpose of your predictions. You can google more about these methods. A method that people generally use is that, for each of the k-folds, they also make predictions for the actual test set and later on take the mean of all the k predictions to generate the final predictions.
Depiction of K-Fold Cross Validation (Image Source: Wikipedia)

GridSearchCV is a method used to tune the hyperparameters of your model (For Example, max_depth and max_features in RandomForest). In this method, you specify a grid of possible parameter values (For Example, max_depth = [5,6,7] and max_features = [10,11,12] etc.). GridSearch will now search for the best set of combination of these set of features that you specified using the k-fold cv approach that I mentioned above i.e. it will train the model using different combinations of the above mentioned features and give you the best combination based on the best k-fold cv score obtained (For Example, Trial1: max_depth = 5 and max_features = 10 and and K-fold CV Accuracy Score Obtained is 80%, Trial2: max_depth=5 and max_features=11 and K-fold CV Accuracy Score Obtained is 85% and so on...) GridSearch is known to be a very slow method of tuning your hyperparameters and you are much better off sticking with RandomSearchCV or the more advanced Bayesian Hyperparameter Optimization methods (you have libraries like skopt and hyperopt in python for this). You can google more about these methods too.  
