We have a classification problem: classify type A tumour from type B tumour. In total we have 50 patient cases (25 A and 25 B cases). We use texture or shape analysis to generate features we can classify. Then how many features we should use to avoid overfitting? 1000? 100? 10? or 5?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This will greatly depend on your problem and the type of features you use.
It is generally not the number of features itself that will decide if you overfit or not but rather the method you use to train your model with. However, overfitting is more prone to happen if your data is sparse unless you do something to prevent it.
My recommendation would be to use as many features as you can and then let the model decide which of these will add to it's general performance.
If you use a linear model such as logistic regression you can use L1-regularization to select which variables are of value during training (preferably using cross-validation or similar to estimate your models true performance). If you use R there is a package called glmnet that can do this for you. The python library scikit-learn can do the same.
An alternative is to evaluate the performance in a step-wise manner adding or removing variables depending on some model metric. But in my experience it is both easier and more flexible to use regularization.
$\begingroup$
$\endgroup$
If you are able to use the same number of features for both test and training data, number of features has not anything to do with overfitting, unless this is a very exceptional case.
If you have the possibility of using such a variable range of features (10 to 1000) then perhaps you should be more concerned about the correlation between your features. You maximize the interpretability and efficiency of your mode, use a "dimensionality-reduction" method to get rid of the features which are not contributing significantly to the classifications.As @While suggested, your model can be used to shrink the dimensionality. Correlation analysis is another approach which is more statistical and less machine learning approach.
Overfitting usually comes with going too far with making your model more and more complex for obtaining higher accuracy on the training set, while resulting in losing the generality of your model, thus memorizing the training set instead of learning it. The number of features use in a model doesn't have anything to do with memorizing the training set.
