I am thinking about doing the following to a data set with $N$ samples and $m$ features

1) Train using semi-supervised learning and cross validate on labeled data using LOO-CV to select the best model.

2) Once we have the best model, eliminate one feature and go again to 1. Search again for the best model.

3) Stop when you have $n < m$ features and the best model

Do this tend to overfit?

Edit: Would it be better if I adjust the model with all the features and then perform backward feature selection only?

  • if you optimize your model in some way, you need a validation step that is outside (= independent) of the optimization. Thus, if you optimize using CV results, a double or nested validation set-up is necessary.

  • While CV can help to reduce overfitting for a given step, the proposed iterative optimization will tend to overfit to your data set.

  • Instead of doing the backward feature selection with the overfitting problem, you could think about e.g. using LASSO regularization which will also produce a feature selection by shrinking coefficients to 0.

  • Semi-supervised models assume that unlabeled and labeled data come from the same distribution. This is often violated [Berget, I. & Næs, T. Using unclassified observations for improving classifiers, J Chemom, 18, 103-111 (2004). DOI10.1002/cem.857]
    You need to be more careful if you optimize your model: the semi-supervised approach in itself already increases the risk of overfitting as unlabeled data essentially cannot correct the model, it can only refine estimates of where data points are expected.

  • $\begingroup$ Could it be better then to: 1) Build a classification model with manifold using 5-fold CV on the labeled set 2) Apply feature selection and select n features 3) Build a classification model with that n features and validate it. Does that work better? $\endgroup$ – KoTy Nov 25 '14 at 18:35
  • $\begingroup$ In the post i propose taking all the subsets with n-1 features and searching which has the best accuracy searching in each subset for the best model, then form the best subset, taking n-2 features and so on Wouldn't that work? $\endgroup$ – KoTy Nov 25 '14 at 18:37
  • 1
    $\begingroup$ The problem with such optizmization is the large number of model comparisons. Due to the finite number of test cases you have an uncertainty in measuring the actual performance of each model. So the question is, is the detected difference true or is it caused by an accidentally nicely fitting test set? The more comparisons you have, the higher the risk of picking accidentally nice looking combinations of hyperparameters and test cases. I.e. you overfit to your test set. Yes, this risk is lowered by cross validating vs. a single split, but it increases with increasing no of comparisons. $\endgroup$ – cbeleites Nov 26 '14 at 15:28

CV results in a series of models and it's not clear why you would use CV to select the model. I think a better approach is to decide how you want to develop the model, fit the model in the whole dataset, and use the optimism bootstrap to unbiasedly validate that model using resampling.

  • $\begingroup$ Thanks for the answer. 2 things: 1) I was proposing using Leave-one-out (LOO) CV, due to that K-fold CV may no generate i.i.d. sets as in semi-supervised label data is scarce. However bagging may be better. $\endgroup$ – KoTy Nov 24 '14 at 20:40
  • $\begingroup$ 2) I was proposing selecting a subset of features (backward style) and adjusting the model parameters so that the error is minimum. Would it be better if I adjust the model with all the features and then perform backward feature selection only? $\endgroup$ – KoTy Nov 24 '14 at 20:42
  • $\begingroup$ It is not clear why CV is relevant to the model building part of your project. But I can't understand you last comment. Please re-state. $\endgroup$ – Frank Harrell Nov 24 '14 at 21:14
  • $\begingroup$ I mean, I am using manifold regularization (semi-supervised learning) to build a classifier. It has various free parameters. I use CV to select the best free parameters. Then I select a subset of feature and refit the model using CV again $\endgroup$ – KoTy Nov 24 '14 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.