For any classification/recommendation algorithm how should I approach to decide from a given list of features which is the strongest predictor? Is this known as feature engineering? what analysis should one do on the data like plotting graph etc. to determine the good set of features?

  • $\begingroup$ This is a very broad question and the subject of an entire subset of statistical analysis. One answer will for sure not cover all relevant factors. Feature engineering is creating new variables from existing ones, you are instead referring to feature selection. If you would like a quick and dirty tip, I would say consider using Lasso regularization. You get good predictions and as a side effect, feature selection through shrinkage. $\endgroup$ – candles_and_oranges Oct 13 '16 at 18:30
  • $\begingroup$ This is algorithm dependent, and as such it's too broad to conform to StackExchange format, imo. $\endgroup$ – Firebug Oct 13 '16 at 23:03

What you are asking about is called either feature selection or feature importance depending on the context. Feature selection would be if you want to subset your features either before or during modeling to improve performance. Feature importance would be if after you create your model, you want to look back and see which features were the model helpful in your final classification. Either way, it is highly dependent on the method your are using for your classifier. For instance, a feature with a parabolic relationship with the classes (ie very high or very low values are one class but values in the middle are another) won't be a very strong predictor in a linear based model (ex. GLM or regularized GLM), but could be highly predictive for models that allow for non linear relationships (ex. SVMs).

If you are interested in feature importance, you may be able to determine this from the model itself. For example, in a GLM model, if you normalized your features before modeling, the feature with the largest coefficient will be the strongest predictor. Random forest can also give feature importance based on the number of trees and the location of the feature in those trees, but it is more complicated than in GLM, though most implementations will offer a method to calculate feature importance for you.

If you are interested in feature selection, there are a plethora of methods available. Using regularization in your model (ex. lasso) is one option where the model itself has an internal penalty on the coefficient values so it will self select variables by bringing their coefficient to zero to escape the penalty. Methods for selection before modeling are even more diverse. A simple one is to look at the correlation between the feature and the class and just choose the variables with the highest correlation. Another option, called Recursive Feature Elimination, uses set aside training data to first make a model with all the features and then uses variable importance as described above to rank features. Features are then removed in order of their importance and new models are made to see what is the optimal subset of features to use. Getting even more complicated, you can use things like genetic algorithms to select the features that optimize a particular model. These are just a sampling of examples, there are many more methods available. All of these methods have pros and cons in terms of time to run (correlations are quick, genetic algorithms are not) and flexibility (correlations only look at linear relations ships, genetic algorithms are tuned to the model).

In short, there is no one answer here. You need to think about why exactly you need to find the strongest predictor, what kinds of methods you are using for modeling, what are the relationships between the predictors and the outcome, and what are the relationships between the features and each other (ie if two features are highly correlated, something like lasso may throw one out even if it is highly predictive because it is redundant with another feature). When you get a better idea of what you want, post again and I am sure you will get a more specific solution.

| cite | improve this answer | |
  • $\begingroup$ (+1) It might be useful to provide some links to the concepts/methods you discuss (e.g. Q&A's or tags on this site). $\endgroup$ – GeoMatt22 Oct 14 '16 at 3:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.