3 added 207 characters in body
source | link

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

Can I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have changed my parameter space (which is a part of the modeling process) based on knowledge I now have from using the whole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data. Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.  

For CV to give an unbiased estimate, every step of the modeling process must be repeated independently in each fold. Isn't selection of a search space for hyperparameters a "step" in the modeling process?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

Can I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have changed my parameter space (which is a part of the modeling process) based on knowledge I now have from using the whole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data. Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.  

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

Can I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have changed my parameter space (which is a part of the modeling process) based on knowledge I now have from using the whole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data. Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.

For CV to give an unbiased estimate, every step of the modeling process must be repeated independently in each fold. Isn't selection of a search space for hyperparameters a "step" in the modeling process?

2 edited title
source | link

If change Does changing the parameter search space after nested CV, does it introduce optimistic bias to use nested CV on the same dataset?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

ItCan I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have essentially changed my parameter space (which is a part of the modeling process)based based on knowledge I now have from using the test datawhole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data.

Can I just repeat nested CV now and expect a fair estimate of generalization error?

If not, what to do? Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.

If change parameter search space after nested CV, does it introduce optimistic bias to use nested CV on the same dataset?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

It seems that I have essentially changed my parameter space (which is a part of the modeling process)based on knowledge I now have from using the test data, and therefore need to evaluate on a dataset external to my current dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data.

Can I just repeat nested CV now and expect a fair estimate of generalization error?

If not, what to do? Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.

Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

Can I just repeat nested CV now and expect a fair estimate of generalization error? If not, what to do? It seems that I have changed my parameter space (which is a part of the modeling process) based on knowledge I now have from using the whole dataset in nested CV, and therefore need to evaluate on a dataset external to my current dataset rather than using nested CV on the same dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data. Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.

1
source | link

If change parameter search space after nested CV, does it introduce optimistic bias to use nested CV on the same dataset?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space to c:[0.5,1,1.5,2,2.5,3,3.5,4].

It seems that I have essentially changed my parameter space (which is a part of the modeling process)based on knowledge I now have from using the test data, and therefore need to evaluate on a dataset external to my current dataset. I am not "choosing" parameters explicitly based on the test data, but I am allowing for their choice based on the testing data.

Can I just repeat nested CV now and expect a fair estimate of generalization error?

If not, what to do? Should I perform nested CV on a subset of the data to find good parameter spaces and then repeat on the full data? It seems that doing so would still allow the algorithm to see some of the data while choosing parameters and in worst case that "some" of the data randomly finds its way into every test fold upon repeated nested CV.