3
$\begingroup$

I remember reading about a general rule of thumb for choosing number of features. It was something like $\sqrt{\log_2(n)}$, where $n$ is the number of samples or $\log_2(n)$. Does anyone remember the original paper (or book) that describes the formula?

Alternatively, could you kindly describe a good guideline to derive the number of features to use?

Thank you!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Number of features to use for what purpose? What is the context? (In its current state, this question is to broad to answer.) $\endgroup$
    – whuber ♦
    Commented May 19, 2012 at 22:05
  • $\begingroup$ I agree that a specific answer might be hard to come by. However, I opine that the OP is not looking for something specific ("rule of thumbs") - it seems preferably techniques that can be used in a large variety of situations. Perhaps this should be a community wiki? $\endgroup$
    – Andrew
    Commented May 19, 2012 at 22:10

1 Answer 1

Reset to default
2
$\begingroup$

Often when I want to select the number of features, I tend to look at Principal Component Analysis (PCA).

If you consider the different features as the dimensions of your problem, PCA will allow you to create a new set of features (with less dimensions) that preserves most of the information. Each of the new features has an associated eigenvalue that describes the amount of information preserved from the original formulation of the problem.

The more of the new features you use, the more information you have (about the original problem). However, often a few features are enough to give you most of the information that you are seeking and a lot of features give little additional information at all (since they have small eigenvalues).

To identify this cut-off point you can use a scree plot:

Scree Plot

If you notice, after the first 3-4 features (components in this diagram) you get little extra information about the original problem (because the eigenvalues are small). So if you create a new problem with only the first 4 features, you preserve most of the information that you have available while reducing the dimensionality and complexity of your original problem.

HTH!

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you!, I was trying to build a classifier with logistic regression and I was trying to limit myself from using too many features which can lead to overfitting. $\endgroup$
    – andrew
    Commented May 19, 2012 at 22:23
  • $\begingroup$ No problem. Personally I do not agree with the rule of thumb that you mentioned in your question. e.g. A set of 1,000,000 items might be classified well with just 2 features. You should edit your own question to incorporate this new information: perhaps you will get a better answer than mine for your specific problem. $\endgroup$
    – Andrew
    Commented May 19, 2012 at 22:26
  • 1
    $\begingroup$ I think your suggestion is still well-suited for my question. I was asking about choosing the maximum number of feature that has lower chance of overfitting the data, but I forgot to mention them. Also, now I realize that it depends on what algorithm I use, and characteristic of data. $\endgroup$
    – andrew
    Commented May 19, 2012 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.