i am using Keras on a text classification task in RStudio.
I have a very imbalanced binary classification problem where the positive class is only present in about 2% of cases.
If i use down-sampling and only take 2% of the negative cases, i can achieve over 90% accuracy on my validation set with both 90% specificity and 90% sensitivity, however I would like to know if there is a way to use the entire dataset and a metric like F1 or Kappa instead of accuracy to cope with the imbalance.
This is my model currently (used on downsample data):
model <- keras_model_sequential() %>% layer_embedding(input_dim = vocab_size, output_dim = 16) %>% layer_global_average_pooling_1d() %>% layer_dense(units = 64, activation = "relu") %>% layer_dense(units = 1, activation = "sigmoid") %>% compile( optimizer = 'adam', loss = 'binary_crossentropy', metrics = c('accuracy') )
Is there a way i could replace metrics = c('accuracy') with F1 or Kappa, or both Sensitivity and Specificity?