Predictive features with high presence in one class I am doing a logistic regression to predict the outcome of a binary variable, say whether a journal paper gets accepted or not. The independent variable or predictors are all the phrases used in these papers - (unigrams, bigrams, trigrams). One of these phrases has a skewed presence in the 'accepted' class. Including this phrase gives me a classifier with a very high accuracy (more than 90%), while removing this phrase results in accuracy dropping to about 70%. 
My more general (naive) machine learning question is:


*

*Is it advisable to remove such skewed features when doing classification? 

*How do you such handle features which are intending to predict only one class? 

*Is there a method to check skewed presence for every feature and then decide whether to keep it in the model or not?

 A: *

*Is it advisable to remove such skewed features when doing
classification?


*

*In general, no. In various specific cases - yes. (If you hand engineered features, or used something very specific for example). Sometimes a strong feature is just that. (While in other cases it can be overfitting). 
What's the phrase? It might be "Accepted" or "published" for example :). 


*How do you such handle features which are intending to predict only
one class?


*

*Be VERY careful with your "negative set", assuming you're using discriminative classification. (And not 1 class or unsupervised methods). 
How large and "random"/diverse is your data? (Sources, fields, journals, types? )
Are oyu looking at the final text in the journal or the author submission? (See my question on how whether the feature is something like "accepted", "published", etc' ).


*Is there a method to check skewed presence for every feature and then
decide whether to keep it in the model or not?


*

*Best way would be Chi square (plenty of inbuilt methods. eg, sci kit learn), or Mutual Information (entropy, etc'. Maybe mrmr?), or P. correlation.


