I am doing a logistic regression to predict the outcome of a binary variable, say whether a journal paper gets accepted or not. The independent variable or predictors are all the phrases used in these papers - (unigrams, bigrams, trigrams). One of these phrases has a skewed presence in the 'accepted' class. Including this phrase gives me a classifier with a very high accuracy (more than 90%), while removing this phrase results in accuracy dropping to about 70%.
My more general (naive) machine learning question is:
- Is it advisable to remove such skewed features when doing classification?
- How do you such handle features which are intending to predict only one class?
- Is there a method to check skewed presence for every feature and then decide whether to keep it in the model or not?