i am using Keras on a text classification task in RStudio.

I have a very imbalanced binary classification problem where the positive class is only present in about 2% of cases.

If i use down-sampling and only take 2% of the negative cases, i can achieve over 90% accuracy on my validation set with both 90% specificity and 90% sensitivity, however I would like to know if there is a way to use the entire dataset and a metric like F1 or Kappa instead of accuracy to cope with the imbalance.

This is my model currently (used on downsample data):

model <- keras_model_sequential() %>% 
   layer_embedding(input_dim = vocab_size, output_dim = 16) %>%
   layer_global_average_pooling_1d() %>%
   layer_dense(units = 64, activation = "relu") %>% 
   layer_dense(units = 1, activation = "sigmoid") %>% 
      optimizer = 'adam',
      loss = 'binary_crossentropy',
      metrics = c('accuracy')

Is there a way i could replace metrics = c('accuracy') with F1 or Kappa, or both Sensitivity and Specificity?

  • $\begingroup$ Don't know about the metrics question (although I do know that accuracy is a poor metric for logistic regression) but you do have other options for the imbalance problem: stackoverflow.com/questions/52103972/… $\endgroup$ – DWin Mar 20 '19 at 22:13

When dealing with an outcome with a very low prevalence, it is not appropriate to use classifiers but rather to model tendencies, i.e., probabilities. There are several proper accuracy scores for probability forecasts. The issues with classification are discussed in detail here and in other blog articles on that site.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.