I try to solve a binary classification problem.
I have a set of features to build a model. The simplest model just pics a single feature $f$ and optimizes a cutoff $c_f$ to separate the two classes.
Question 1:
Can I think of the feature to select and its corresponding cutoff as of 2 parameters to be optimized and run the whole process in a standard k-fold cross-validation loop?
I.e. I do an exhaustive search of all possible feature/cutoff combinations on the training folds, select the one combination with best performance and evaluate it on the test fold. By averaging over all test folds, I get the estimate for the expected performance on unseen data for a model trained the same way on the whole data set.
Or would I rather need to split feature selection and cutoff optimization already in a nested cross-validation?
Now I want to extend the model to two features $f_1, f_2$ that will be logically combined to define the classification model. So I will fit two cutoffs $c_{f_1}, c_{f_2}$ and predict the class of a case $x$ as $\text{class}(x) = f_1(x) \geq c_{f_1} \land f_2(x) \geq c_{f_2}$.
Again, I would take all possible combinations of two features and their cutoffs, identify the one combination performing best on the training folds, and apply it to the test fold.
Question 2:
Can I think of the number of features $n$ to include (1 or 2) as of a hyperparameter of my model?
For model selection, i.e. optimization of $n$, I would then run a nested cross-validation. The inner CV uses the above described process to identify the best univariate ($n=1$) model on the inner training folds and evaluate its performance on the inner test fold. The same for the bivariate models ($n=2$). Then I would rank them (the top-candidate from $n=1$ and the top-candidate from $n=2$) according to their performance estimate from the inner CV, and select the best one to apply it to the outer test fold.
Then I would finally run the inner CV on the whole data set to get the final model. The average performance on the outer test folds would then serve as a unbiased estimated of the expected performance of this final model on unseen data.
Is this a valid setup?
