Return to Answer

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation. Then build your final predictive model using all the data.

As you have more features than cases, you are very likely to over-fit the data simply by feature selection. It is a bit of a myth that feature selection improves predictive performance, so if that is what you are interested in (rather than identifying the relevant features as an end in itself) then you are probably better off using ridge regression and not performing any feature selection. This will probably give better predictive performance than feature selection, provided the ridge parameter is selected carefully (I use minimisation of Allen's PRESS statistic - i.e. the leave-one-out estimate of the mean-squared error).

For further details, see Ambroise and McLachlan, and my answer to this question.

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation. Then build your final predictive model using all the data.

As you have more features than cases, you are very likely to over-fit the data simply by feature selection. It is a bit of a myth that feature selection should be expected to improve predictive performance, so if that is what you are interested in (rather than identifying the relevant features as an end in itself) then you are probably better off using ridge regression and not performing any feature selection. This will probably give better predictive performance than feature selection, provided the ridge parameter is selected carefully (I use minimisation of Allen's PRESS statistic - i.e. the leave-one-out estimate of the mean-squared error).

For further details, see Ambroise and McLachlan, and my answer to this question.

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation. Then build your final predictive model using all the data.

As you have more features than cases, you are very likely to over-fit the data simply by feature selection. It is a bit of a myth that feature selection improves predictive performance, so if that is what you are interested in (rather than identifying the relevant features as an end in itself) then you are probably better off using ridge regression and not performing any feature selection. This will probably give better predictive performance than feature selection, provided the ridge parameter is selected carefully (I use minimisation of Allen's PRESS statistic - i.e. the leave-one-out estimate of the mean-squared error).

For further details, see Ambroise and McLachlan, and my answer to this question.

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation. Then build your final predictive model using all the data.

As you have more features than cases, you are very likely to over-fit the data simply by feature selection. It is a bit of a myth that feature selection improves predictive performance, so if that is what you are interested in (rather than identifying the relevant features as an end in itself) then you are probably better off using ridge regression and not performing any feature selection. This will probably give better predictive performance than feature selection, provided the ridge parameter is selected carefully (I use minimisation of Allen's PRESS statistic - i.e. the leave-one-out estimate of the mean-squared error).

For further details, see Ambroise and McLachlan, and my answer to this question.

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation.

For further details, see Ambroise and McLachlan, and my answer to this question.

The procedure you are using will result in optimistically biased performance estimates, because you use the data from the test set used in steps 2 and 3 to decide which features used in step 1. Repeating the exercise reduces the variance of the performance estimate, not the bias, so the bias will not average out. To get an unbiased performance estimate, the test data must not be used in any way to make choices about the model, including feature selection.

A better approach is to use nested cross-validation, so that the outer cross-validation provides an estimate of the performance obtainable using a method of constructing the model (including feature selection) and the inner cross-validation is used to select the features independently in each fold of the outer cross-validation. Then build your final predictive model using all the data.

As you have more features than cases, you are very likely to over-fit the data simply by feature selection. It is a bit of a myth that feature selection improves predictive performance, so if that is what you are interested in (rather than identifying the relevant features as an end in itself) then you are probably better off using ridge regression and not performing any feature selection. This will probably give better predictive performance than feature selection, provided the ridge parameter is selected carefully (I use minimisation of Allen's PRESS statistic - i.e. the leave-one-out estimate of the mean-squared error).

For further details, see Ambroise and McLachlan, and my answer to this question.