Trying to get better precision/recall for both classes ... any tips?
- I have heterogeneous features [a few num vars, a few cat vars, and 2 text vars]
- Target is a binary classification w/ class imbalance [about 85% class 1 and 15% class 0]
- Don't have much training data [only around 17K rows]
Here is my pipeline:
cat_transformer = Pipeline(steps=[
('cat_imputer', SimpleImputer(strategy='constant', fill_value='missing')),
('cat_ohe', OneHotEncoder(handle_unknown='ignore'))])
num_transformer = Pipeline(steps=[
('num_imputer', SimpleImputer(strategy='constant', fill_value=0)),
('num_scaler', StandardScaler())])
text_transformer_0 = Pipeline(steps=[
('text_bow', CountVectorizer(lowercase=True,\
token_pattern=SPLIT_PATTERN,\
stop_words=stopwords))])
# SelectKBest()
# TruncatedSVD()
text_transformer_1 = Pipeline(steps=[
('text_bow', CountVectorizer(lowercase=True,\
token_pattern=SPLIT_PATTERN,\
stop_words=stopwords))])
# SelectKBest()
# TruncatedSVD()
FE = ColumnTransformer(
transformers=[
('cat', cat_transformer, CAT_FEATURES),
('num', num_transformer, NUM_FEATURES),
('text0', text_transformer_0, TEXT_FEATURES[0]),
('text1', text_transformer_1, TEXT_FEATURES[1])])
pipe = Pipeline(steps=[('feature_engineer', FE),
("scales", MaxAbsScaler()),
('rand_forest', RandomForestClassifier(n_jobs=-1, class_weight='balanced'))])
random_grid = {"rand_forest__max_depth": [3, 10, 100, None],\
"rand_forest__n_estimators": sp_randint(10, 100),\
"rand_forest__max_features": ["auto", "sqrt", "log2", None],\
"rand_forest__bootstrap": [True, False],\
"rand_forest__criterion": ["gini", "entropy"]}
strat_shuffle_fold = StratifiedKFold(n_splits=5,\
random_state=123,\
shuffle=True)
cv_train = RandomizedSearchCV(pipe, param_distributions=random_grid, cv=strat_shuffle_fold)
cv_train.fit(X_train, y_train)
from sklearn.metrics import classification_report, confusion_matrix
preds = cv_train.predict(X_test)
print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))
On average based on many different combinations of attempts via classification report i am getting around:
- class 1 => approx. 95% precision; 98% recall
- class 0 => approx. 80-85% precision; 57-66% recall
When i perform stratified k-fold shuffles and added class_weight='balanced" i can get to 66% recall, but would like to get around 75%-80%
questions:
- Any other feature engineering techniques i can do to improve predicting class 0? [have tried different things on text like TFIDF, Hashing Trick, selectKBest, SVD(), and maxAbsScaler() on all features]
- Any other algorithms i should try? [have only tried random forest classifier]
- Is low recall a big deal?
- Mostly have been just "plugging and playing" ... anything obvious i am missing?
- Would applying over-sampling help? if so, how can that be done in python / sklearn?
Any help would be much appreciated!