What is accepted practice for avoiding optimistic bias when selecting a model family after hyperparameter tuning? This is an extension of a previous question:
How to avoid overfitting bias when both hyperparameter tuning and model selecting?
...which provided some options for the question at hand, but now I would like to pivot to knowing what is accepted practice or rule of thumb.
In short, say we do hyperparameter tuning on multiple ML model families. The following selection step of choosing the model family itself provides another opportunity for optimistic bias. This could be resolved by some of the strategies noted in the link above.
Noting the previous discussion, are there accepted rules of thumb (or research) on when said strategies are important? For instance, if just optimizing two model families, is it generally safe to ignore the concern and pick the model family in the train split score (or perhaps even the test split)? Or is there a certain n number of model families at which this becomes a danger and tripple-nesting or gridsearch modifications of some kind is needed?
 A: Model selection can often be seen - in a broad sense - as part of hyperparameter tuning. 
An example: let's say that we have to solve a simple regression problem, and we want to use some sort of linear model. In this scenario we could choose linear models with or without a polynomial expansion, with or without a L1 or L2 regularization term.
One could see the problem as follows:
 - A model selection between L1 regularized, L2 regularized, and not regularized models
 - Hyperparameter tuning to define the order of polynomial and (if present) regularization terms

Alternatively, it can be seen as just one large model (like an Elastic Net) with polynomial expansion, and everything becomes a hyperparameter. 
This example is particularly trivial, because hyperparameter tuning and model selection are directly linked by the regularization coefficient (if we put them to 0, we go into the non-regularized model). However this can always be done, and model selection can be seen as part of hyperparameter tuning, with the set of hyperparameters being conditional on the choice of the first hyperparameter (the model). 
This might sound weird, but conditional hyperparameters are very common: for example, the hyperparameter number of units in the 3rd layer of my Neural Net is conditional to the hyperparameter depth of my neural network being larger than 2.

So finally what's the best practice?
 I would say that it depends on the amount of data you have and the amount of hyperparameters you use. The less data and the more hyperparameters, the more bias you will have. If you have enough data, doing a nested cross validation is most likely overkill and you should stick to a flat CV, as mentioned in the post above.  If however you are testing a huge amount of parameter configurations or different models, it might be worth either using nested cross validation, or using a separate validation set to check the results of your cross validation. This can mean either doing hyperparameter tuning via cross validation, and model selection on a separate set, or even simply re-scoring your best performing models and configurations after you filtered them via cross val. 
A: I wrote a paper (with a co-author) on these topics (nested vs non nested cross-validation).
Please see it at https://arxiv.org/abs/1809.09446
TLDR: for practical purposes we advise against using nested cross validation. Although there is indeed a positive/overfitting bias in performing a flat search (as opposed to nested) the difference is below what we consider a practical threshold of irrelevance (which is compatible with others proposal for this threshold).
But this is a practical proposal, when your goal is to select the best model+hyperparameters for your problem. If you have a scientific problem of showing that your algorithm is better than the competition, you should perform the nested CV.
