If you are really worried about unaccounted "researcher degrees of freedom", I suspect the best you can do is to automate your model construction procedure (including feature selection, hyper-parameter optimisation etc.), so that the researcher is not making any choices after seeing the data. You can then use cross-validation provided you perform the whole model construction procedure independently in each fold.

In practice, it is often best to simply acknowledge there are unaccounted degrees of freedom, and make an effort not to over-analyse the data. While AutoML has become a hot topic in machine learning, I suspect "CyborgML" where the operator becomes an implicit and unaccounted part of the machine learning procedure is likely to be a significant issue, especially in Deep Learning, where models are computationally expensive to fit. This ought to be less of a problem for more classical statistics.

If you are really worried about unaccounted "researcher degrees of freedom", I suspect the best you can do is to automate your model construction procedure (including feature selection, hyper-parameter optimisation etc.), so that the researcher is not making any choices after seeing the data. You can then use cross-validation provided you perform the whole model construction procedure independently in each fold.

In practice, it is often best to simply acknowledge there are unaccounted degrees of freedom, and make an effort not to over-analyse the data. While AutoML has become a hot topic in machine learning, I suspect "CyborgML", where the operator becomes an implicit and unaccounted part of the machine learning procedure, is likely to be a significant issue for the foreseeable future, especially in Deep Learning, where models are computationally expensive to fit. This ought to be less of a problem for more classical statistics.

If you are really worried about unaccounted "researcher degrees of freedom", I suspect the best you can do is to automate your model construction procedure (including feature selection, hyper-parameter optimisation etc.). You can then use cross-validation provided you perform the whole model construction procedure independently in each fold.

In practice, it is often best to simply acknowledge there are unaccounted degrees of freedom, and make an effort not to over-analyse the data. While AutoML has become a hot topic in machine learning, I suspect "CyborgML" where the operator becomes an implicit and unaccounted part of the machine learning procedure is likely to be a significant issue, especially in Deep Learning, where models are computationally expensive to fit. This ought to be less of a problem for more classical statistics.

If you are really worried about unaccounted "researcher degrees of freedom", I suspect the best you can do is to automate your model construction procedure (including feature selection, hyper-parameter optimisation etc.), so that the researcher is not making any choices after seeing the data. You can then use cross-validation provided you perform the whole model construction procedure independently in each fold.

In practice, it is often best to simply acknowledge there are unaccounted degrees of freedom, and make an effort not to over-analyse the data. While AutoML has become a hot topic in machine learning, I suspect "CyborgML" where the operator becomes an implicit and unaccounted part of the machine learning procedure is likely to be a significant issue, especially in Deep Learning, where models are computationally expensive to fit. This ought to be less of a problem for more classical statistics.