Choosing the number of features I remember reading about a general rule of thumb for choosing number of features. It was something like $\sqrt{\log_2(n)}$, where $n$ is the number of samples or $\log_2(n)$. Does anyone remember the original paper (or book) that describes the formula?
Alternatively, could you kindly describe a good guideline to derive the number of features to use?
Thank you!
 A: Often when I want to select the number of features, I tend to look at Principal Component Analysis (PCA).
If you consider the different features as the dimensions of your problem, PCA will allow you to create a new set of features (with less dimensions) that preserves most of the information.   Each of the new features has an associated eigenvalue that describes the amount of information preserved from the original formulation of the problem.
The more of the new features you use, the more information you have (about the original problem).  However, often a few features are enough to give you most of the information that you are seeking and a lot of features give little additional information at all (since they have small eigenvalues).
To identify this cut-off point you can use a scree plot:

If you notice, after the first 3-4 features (components in this diagram) you get little extra information about the original problem (because the eigenvalues are small).  So if you create a new problem with only the first 4 features, you preserve most of the information that you have available while reducing the dimensionality and complexity of your original problem.
HTH!
