How to select the optimal combination of input variables for supervised learning model? While working on a specific assignment, where the goal was to predict the outcome class, I tried several models (neural networks, logistic regression and my custom prediction model) with several combinations of 3 available input variables.
What I found interesting is that the models with two variables outperformed the models with 3 input variables and I would be glad to find some theoretical explanation for such outcome.
Does anyone know, whether there are any sistematic approaches to select the optimal set of input variables? If the input variables are correlated among themself, does this reduce the accuracy of prediction model?
Thank you.
 A: I can give you one possible reason why two variables can sometimes perform better than three.  It could be that a model with three variables is overfitting.  One way to decrease overfitting is to reduce the complexity of the model (eg. using two variables instead of three).  This way the model will generalize better to unseen test data and perform better.  There are ways to see if there is an overfitting problem (eg. comparing training error to test error).
In terms of systematic ways to select the optimal set of input variables, this is called feature selection.  One such approach is to try every single possible combination of input variables and see which combination performs the best.  This can be intractable if you have a large number of possible features so sometimes approximation techniques are used (eg. greedy search).
The impact of correlation among input variables depends on the prediction algorithm that you are using.  Every model has its own inductive bias.  If you are worried about redundancy within your input variables you should try dimensionality reduction (eg. PCA) which will reduce your input variables to only the most informative (i.e. variables that contribute most of the variation within your dataset). 
A: There are many possible systematic approaches to choosing your features. The approaches are often called feature selection, feature reduction, feature extraction, dimensionality reduction etc. etc. (really there are a ridiculous number of names for it). Methods used to choose the features can include greedy approaches, evolutionary algorithms, specialist domain knowledge, more mathematically grounded approaches like PCA, etc. If you have a particular problem in mind then I would suggest reading up on some of the literature and getting an idea of the approach that is most suitable for you. For example, some methods will cause the meaning of your original features to be lost (e.g. through combining features), and if this is not wanted then you may only want to consider selecting a subset of your original features.
As to the theoretical explanation of why fewer features may be better, you might want to have a look at information on the Hughes effect/Hughes phenomenon/peaking paradox (see e.g. here or Wikipedia). Basically, the performance of a classifier improves with addition of features until a certain (problem dependent) point. After that adding more features can (possibly) be detrimental to the performance of the classifier. Normally this is an issue for higher dimensional problems, but I though it might be of interest to you anyway.
A: Through Data Mining, we can identify best combinations if large data.
Otherwise go for step wise regression models.
