Feature Selection Methodology I have a question regarding methodology in Machine Learning.
Assume we have a classification problem, the size of training set is 100 every instance in training set is represented by 200 features. In order to eliminate meaningless features we evaluate Information Gain / Mutual Information of every feature and eliminate those with low mutual information. Then in order to evaluate the model we use 10-fold cross validation.
The question is after all these manipulation whether the calculated evaluation can be used as real evaluation, in other words, are we allowed to perform all these manipulation and get reasonable model with reasonable evaluation.
Ideas: Initially the entire process looked for me reasonable, but then I though if I am allowed to eliminate features based on the Infogain of train set or better to do it on validation set, on the other hand, in any case the trained model has access only to train set, so if features has low infogain it will have a minor weight in the model.
I would appreciate if you could share your thoughts.
 A: Generally speaking, the problem of feature selection is computationally very hard to solve. What you are trying to do is to find an optimal subset out of $2^n$  possible sets (where $n$ is the number of features), which is an NP hard problem. As n grows, this problem is not solvable exactly due to it's computational complexity. 
That said, there are ways to find approximate solutions. Using a simple greedy search approach such as forward-, or backward step-wise selection as you describe, will not guarantee an optimal solution and are not even stable, but will in some cases give you a reasonably good approximation. So the answer to your question (if I understand it correctly) is no. It is not viable to do this manipulation to get an optimal model. It is however common to use these methods to find a good model, even if it is not optimal. 
If you are interested, there are other approaches to this problem. One is for example using L1-regularization (LASSO). As the LASSO uses an absolute shrinkage term it is possible for $\beta$-values to be pushed down to zero, essentially performing a subset selection. I would recommend Regression modeling strategies by Harrell and Elements of 
Statistical Learning by Hastie et al. for further reference on the subject. 
