CV for Model selection - Why is significance testing not needed? Say we have a model and some hyper-parameter with L values, and our goal is model selection. A k-fold CV outputs L accuracies (each accuracy is an average over K values). The best model corresponds to the highest accuracy.
When comparing the L accuracies in order to select the best model, why don't we do any significance testing?
My guess is that there is no cost in wrongly rejecting the null hypothesis (== saying one model is better than the other while they are equivalent), therefore there is no harm in selecting the model with the highest accuracy. The worse that can happen is that all L models are equivalent, but even if that's the case we still want to select one of them, so there is no harm in selecting the highest accuracy (it's as arbitrary as anything else)
 A: There isn't usually a genuine statistical distinction between hyper-parameters and parameters, it is normally just a matter of computational convenience.  There is a computationally efficient method of optimising the parameters for fixed values of the hyper-parameters, so it is sensible to use it.  We call the parameters for which there is no computationally efficient algorithm "hyper-parameters" and use cross-validation to tune them instead, which is often computationally expensive.  So the main reason we don't need statistical hypothesis tests for hyper-parameters is that we don't usually need them for parameters either.
However, it is possible to over-fit the cross-validation error when tuning the hyper-parameters, just as you can over-fit the training data when optimising the parameters.  This can result in a model that under-fits the training data as well as over-fitting it.  There are a number of practical solutions, such as to regularise the hyper-parameters, another is to tune the hyper-parameters and then backtrack to the point where the cross-validation performance is not statistically distinguishable from that of the best set of hyper-parameters, so you can use NHSTs in model selection if you want to (sort of "early stopping").
A: Summary: IMHO, it would be good to do significance testing for model selection. (But as @Christian Hennig already pointed out, it's actually not so easy to do)
Or rather: we'd need to get a far more realistic idea of the random uncertainty our hyperparameter tuning is subject to.
Whether that takes the shape of significance testing, Bayesian model averaging, or more sophisticated cross validation, any other for or a combination thereof I don't care.
But I do think we have overall a serious problem with overfitting. (Which I think also has strong links to the reproducibility crisis in the life sciences)

@Dikran Marsupial points out that hyperparameters are not inherently different from other parameters of a model. The distinction is mainly a question of implementation convenience, leaving parameters that are difficult to fit correctly in the general case to the care of the user.
This is also basically the approach that has been used for many decades in classical statistics: e.g. for a linear model, we know how to calculate a least squares fit on given data. It is the genuine task of the one who models some data to specify the model terms, including model complexity, feature selection (which variates to include/drop) and feature construction (e.g. polynomial terms).
to be continued
