So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be selected as a constant, then let my feature selection method pick whatever N features it wants to for each fold of cross validation.
E.g. Set N = 1, evaluate CV performance for all classifiers I tested with a single feature Set N = 2, Repeat with 2 features... Select the N with the best CV performance.
So I now have K sets of N features each. To train a final model on all of the data, I need to settle on the "best" N features. Do I select (1) the pair of features which appeared together the most out of K folds, or do I select (2) the top N features which were selected the most out of all K folds?
You can think of (2) as a histogram of all the features which were ever selected in all K folds and each fold provides a vote if it picked that feature. (1) would be like (2) except each bin is a pair of features, rather than a single feature.