# How to spot and remove unimportant columns in a dataframe?

I have a pandas dataframe with several columns, for example let's say:

   A   B   C   D(labels)
45  88  44  0
62  34   2  1
85  65  11  1
74  43  42  1
90  38  34  0
...
0  94  45  1
58  23  23  0


How can I detect which columns are useless and which columns are useful to a machine learning model?. I all ready tried several methods like PCA, removing features with low variance, univariate feature selection with a chi criteria, etc. However none of them seems to work since the performance of my classifier still is low. I also tried to create more features (add more columns to the feature matrix) and they decreased the performance of my classifier, thus is there anyway to spot which columns are useless?.

• It could also just be that there isn't sufficient information for the classification to perform well. A couple other approaches you could try though would be to find and remove correlated features if they exist, which can through off some methods, or (depending on the dimensions of your data), try something like Lasso regression which will down-weight unimportant features without removing them entirely, or random forests which has the ability to assign an importance score to each feature. – Keith Hughitt Aug 24 '16 at 10:29
• Thanks for the help @KeithHughitt, which is the approach to check the correlation between to columns? – tumbleweed Aug 25 '16 at 6:29

I'm assuming all training/CV/test performance is bad and thereby that the problem is not overfitting. In a nutshell, you then could try the following to meaningfully reduce your features:

• Use feature correlation to reduce correlated features,
• Feature selection techniques such as feature filters and feature wrappers,
• Feature reduction with using techniques like PCA, or
• Models that internally "weight" features themselves.

Things you should consider:

• As @KeithHughitt mentioned, the problem might be that the relation you seek is simply not present in your data. In such a case it might be impossible for models to perform and generalize well. The "one perfect" solution for those cases does not exist, but, as you already mentioned, deriving features (same information but differently processed) and/or adding information (new information, e.g. with recording more features) might help.

• Another explanation for bad predictive performance with big data/many features might be: the feature-target relation is too complex to be represented accordingly by your model (e.g. trying to model circular data with a linear model). In such cases, another option besides adding preproceesing/feature derivation would be to employ more complex models. But those usually come at the cost of increased calculation power, such as with deep learning.

• I am using gradient tree boosting and I all ready used PCA, SVD and other dimentionality reduction methods. Could you provide some approaches to model feature target relation?.... Do you think Fp-Growth or an association rule algorithm can work for this case? – tumbleweed Aug 25 '16 at 6:43