How to get hyper parameters in nested cross validation? I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation:


*

*Nested cross validation for model selection

*Model selection and cross-validation: The right way
To explain my confusion, let me try to walk through the model selection with nested cross validation method step by step.


*

*Create an outer CV loop with K-Fold. This will be used to estimate the performance of the hyper-parameters that "won" each inner CV loops.

*Use GridSearchCV to create an inner CV loop where in each inner loop, GSCV goes through all possible combinations of the parameter space and comes up with the best set of parameters. 

*After GSCV found the best parameters in the inner loop, it is tested with the test set in the outer loop to get an estimation of performance.

*The outer loop then updates to the next fold as the test set and the rest as training set, and 1-3 repeats. Total possible "winning" parameters are number of folds designated in the outer loop. So if the outer loop is 5 folds, then you will have a performance estimation of an algorithm with 5 different sets of hyper parameters, NOT the performance of one particular set of hyper parameters.


This approach is illustrated on SKLearn's example page:
http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html
Question:
After 4., how do you determine which hyper parameters worked the best? I understand that you want to train your algorithm (e.g. Logistic Regression, Random Forest, etc.) with the COMPLETE data set at the end. But how do you determine which hyper parameters worked the best in your nested cross validation? My understanding is that for each inner loop, a different set of hyper parameters will win. And for the outer loop, you are getting an estimation of your GridSearchCV performance, but you are not getting any one particular set of hyper parameters. So, in the final model creation, how do you know what hyper parameters to use? That's the missing logic I have trouble understanding from other treads. 
Thank you in advance for any tips, especially if @Dikran Marsupial and @cbeleites can chime in!
Edit: If you can, please in your answer use terms like "algorithm" and "hyper parameters". I think one source of confusion for me is when people use the term "model" or "model selection". I get confused whether they are talking about selecting which algorithm to use or what hyper parameters to use.
Edit 2: I have created a notebook that shows two ways of doing nested cross validation. First way is the one shown in the SKLearn example, and another longer way is one that I wrote. The way shown in SKLearn doesn't expose the "winning" hyperparameters, but my longer way does. But the question remains the same. After I have completed the nested cross validation, even with the hyperparameters exposed, what do I do now? As you can see from the hyperparameters at the end of the notebook, they vary quite a bit.
 A: I have read your question and the answer above 2 times (1st time 3 month ago). I'm interested and also want to find the absolute appropriate way to do cross-validation for my data. After a lot of thinking & reading, it seems that I find the holes and here is my fix:

To explain my confusion, let me try to walk through the model selection with nested cross validation method step by step.

*

*Create an outer CV loop with K-Fold. This will be used to estimate the performance of the hyper-parameters that "won" each inner CV loops.

*Use GridSearchCV to create an inner CV loop where in each inner loop, GSCV goes through all possible combinations of the parameter space and comes up with the best set of parameters. (Note: here I assume: data for inner loop = training data for outer loop. You may ask: why ? Answer: https://stackoverflow.com/questions/42228735/scikit-learn-gridsearchcv-with-multiple-repetitions/42230764#42230764 read the answer section by Vivek Kumar step 4)

*After GSCV found the "best parameters" in the inner loop (let's call it inner winner), it is tested with the test set in the outer loop to get an estimation of performance (let's call it outer_fold_score1).

*The outer loop then updates to the next fold as the test set and the rest as training set (to evaluate the "inner winner" on outer loop), the "inner winner" is tested again with the new test set (outer_fold_score2). Then again the outer loop updates to the next fold until the loop is completed. Scores from each fold (outer_fold_score 1,2..) will be average to get the score of the "inner winner" for the outer loop (outer_score)

*The outer loop then updates to the next fold as the test set and the rest as training set (to find the next "inner winner" , and 1- 4 repeats (note that when we repeat 1 we don't create the new K-fold but we use the same outer Kfold every time). With each of 1-4 cycle, we get a "best parameters" (or "inner winner") with an outer_score. The one with the best outer_score will be the winner of the winners


Reasoning:

*

*Basically your question concerns that there are many "winning parameters" for the outer loop. The thing is u didn't complete the outer loop to evaluate and find the "outer winner". Your 4th step only evaluate the "inner winner" in 1 fold in the outer loop, but u didn't "loop it". Therefore I need to replace it with my 4th step - "loop" the evaluation step in the outer loop and get the outer score (by averaging)

*Your 5th step did do the "looping" job in the outer loop, but it's just for building another "inner winner". It didn't loop the "evaluation" of this "inner winner" in the outer loop

A: You do not use nested cross validation to select the hyper parameters of the algorithm, this method is used for estimating the generalisation error of your model building procedure. Where by model building procedure I intend all the steps you applied to reach the final model you are going to be using on field.
A model building procedure could be composed by the rules you applied to decide which preprocessing to apply to the data, which feature to use and finally which hyper parameters to use. Think of this as a sort of "meta-algorithm" that receives as input a particular dataset $D$ and produces as output an "algorithm", composed of a fixed set of preprocessing transformations, features and finally hyper parameters values. 
For example let's say you have  $X , y$ as design matrix and target and you want to train a classifier:
 1. that uses only the first $x$ features in $X$ that have highest correlation with $y$.
 2. you choose the hyper parameters values by minimisation of a 10-fold cross validation error estimate.  
If you apply these two step to a particular pair of $X', y'$ you will obtain a specific algorithm with a definite set of features and fixed hyper parameters which will not necessarily be the same of the one you would obtain for $X, y$ although the model building procedure would be identical i.e.: steps 1 +  2 which are not tied to any particular dataset.
Let's say you did all of the above without having split your data into train-test because you have a small dataset, how do you estimate the generalisation error of the classifier you just created? Can you use the best error you found in the cross validation at step 2 ?
No, the first big problem is in step 1 where you use all the data to select the features to use. Therefore even when you do the cross validation in step 2, the features will already have seen and remember some information present in the test fold at every cross validation run. The result will be an overly optimistic estimate of the test error and this is called feature selection bias. To account for this in your estimation you would need to put the feature selection step inside the cross validation loop of step 2.
Ok, now are we good? Is the best error found in the cross validation with feature selection step inside the loop a fair estimate of the generalisation error?
In theory the answer is still no, the problem is that your hyper parameters have been chosen to minimise the cross validation error on the specific dataset at your disposal, so in a certain sense you are fitting the hyper parameters to your data with the risk of over-fitting them , and this is called model selection bias.  But is this a concern in practice? It depends by the specific application: it is likely to become more severe, as overfitting in training, when dataset is small and the number of hyper parameters to be tuned is relatively large. To account for this when estimating the generalisation error you would apply a nested cross validation as you described, that would then give you a correct estimate of your generalisation error.
Finally to answer your last question, after having a fair estimate of your “model building procedure” generalisation error with a nested cross validation, you would simply apply the procedure (step 1+2) to your entire dataset obtaining a model with a fixed set of feature and set hyper parameters values, but keeping in mind that the error we expect this model to have on unseen data is the nested cross validation estimate. 
