I'm a bit confused about the k-fold cross validation (inner and external) done for the model performance evaluation.
I've read that when you are trying to validate your model, you need to do it without leaking info from the test set to the training set. Data leaking is common when you are doing feature selection on the whole dataset instead of the training set alone (the same goes for the feature standardization) for example.
But how do you prevent that when you are "learning" your best hyperparameters for your model? For example the number of layers of your neural net which performs better on your problem?
I need to "inner" cross validate them in my cross validation task? Or is it irrelevant to select them by the external cross validation? For example by iterating the parameters on 10-fold cross validation alone, and after that picking the best performing ones?
Essentially, need I to validate my process of hyperparameters selection to avoid some kind of bias? I know pretty well that the parameters search (e.g. weights of your neural net) needs to be done in your training set alone, but is it the same for your hyperparameters?