I try to solve a binary classification problem.

I have a set of features to build a model. The simplest model just pics a single feature $f$ and optimizes a cutoff $c_f$ to separate the two classes.

Question 1:
Can I think of the feature to select and its corresponding cutoff as of 2 parameters to be optimized and run the whole process in a standard k-fold cross-validation loop?
I.e. I do an exhaustive search of all possible feature/cutoff combinations on the training folds, select the one combination with best performance and evaluate it on the test fold. By averaging over all test folds, I get the estimate for the expected performance on unseen data for a model trained the same way on the whole data set.
Or would I rather need to split feature selection and cutoff optimization already in a nested cross-validation?

Now I want to extend the model to two features $f_1, f_2$ that will be logically combined to define the classification model. So I will fit two cutoffs $c_{f_1}, c_{f_2}$ and predict the class of a case $x$ as $\text{class}(x) = f_1(x) \geq c_{f_1} \land f_2(x) \geq c_{f_2}$.
Again, I would take all possible combinations of two features and their cutoffs, identify the one combination performing best on the training folds, and apply it to the test fold.

Question 2:
Can I think of the number of features $n$ to include (1 or 2) as of a hyperparameter of my model?

For model selection, i.e. optimization of $n$, I would then run a nested cross-validation. The inner CV uses the above described process to identify the best univariate ($n=1$) model on the inner training folds and evaluate its performance on the inner test fold. The same for the bivariate models ($n=2$). Then I would rank them (the top-candidate from $n=1$ and the top-candidate from $n=2$) according to their performance estimate from the inner CV, and select the best one to apply it to the outer test fold.

Then I would finally run the inner CV on the whole data set to get the final model. The average performance on the outer test folds would then serve as a unbiased estimated of the expected performance of this final model on unseen data.

Is this a valid setup?


1 Answer 1


Your validation strategies are valid.

Essentially cross validation does not estimate the error of models, but the method to build them. So if you encompass all steps in each fold, your estimation is unbiased.

In your case, it shows that you want to perform feature selection in the 1st question, and feature selection as well as model selection in the 2nd question. Because you independently perform these tasks in each fold, your estimation will be unbiased.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.