According to the approach outlined here. I should split the dataset into training set and an independent test set using stratified cross-validation
. This is the hold out way of splitting the dataset using stratified approach.
Using the training set, I will do k fold cross-validation
for hyperparameter tuning and model selection. I have the following questions for which I could not find answers from the document (I don't have access to the book by the author of the blog). Shall be grateful for help.
Question 1) Is this approach of two way split and model selection using k fold cross-validation
known as the nested cross-validation?
Question 2) Considering an imbalanced dataset with 80 examples from class 0 records and 20 examples belonging to class 1 records.Can somebody please explain with a simple example what is the meaning of ensuring that each fold is representative of all strata of the data? What the output of stratification will be for this example? Is it the same as with and without replacement?
Question 3) During model selection using k fold cross-validation
, should the k folds be obtained using stratified cross validation again even if the dataset had been split using stratified cross-validation into training and an independent test set?