I would like to run a 10-fold cross validation on a number of different feature selection tools. For some tools, you can specify k-fold in the Python module (i.e., LassoLarsCV(cv=10)), but others it is not clear how to implement the cross-validation.
Let's assume, I divided my data into 10 random splits and run the feature selection in each fold. Doing so, there will be some set of variables (many same ones as well as new ones) in each fold. How do you cross-validate these nominal outcomes? They are not means or anything so we can take the average of the 10 fold, but all we have is a different number of variables as a result of validation in each fold. In other words, how can I validate the ideal set of variables in cross-validation procedure? Taking the features that are consistently found in each fold?