I'm trying to evaluate the probability of a rare occurrence. My training data is a binary 1/0 for output and a TFIDF Vector of words for input.
Which seems to lend itself to a regression, I've been using xgboost's XGBRegressor with decent results but something is confusing me.
The feature_importances for XGBRegessor seem to pick ngrams which seem to me not that important.
When I ran the exact same test but instead used XGBClassifier the important phrases it chose made more sense to me.
The results of sklearns
roc_auc score indicate similarly, it ranks the classifier as better than the regressor. Which I don't understand.
Can I compare the
roc_auc score of a classifier and regressor as apples to apples?
Does it make sense that a classifier could perform better at predicting a probability than a regressor?