0
$\begingroup$

This question already has an answer here:

I have a data set with a binary response variable, about 30,000 observations of 8 features, some are continuous and some are categorical.

This is an imbalanced data set, the ratio of negatives to positives is about 5:1. So the null accuracy (always predicting negatives) is ~84%.

I know that for imbalanced data sets accuracy is usually not a good metric. But in my context a high accuracy is desirable and because the imbalance is not extreme I think it is possible to improve. I would like to shoot for at least 90% accuracy.

I have tried various feature engineering techniques and machine learning models but I am not even able to hit 86%. For example, decision trees, logistic regression and random forests all give about 85.6-8% accuracy. I used cross-validation for finetuning hyperparameters and checked training accuracy to make sure there was no overfitting.

What could be the reasons for getting such a marginal improvement in accuracy over a dumb model?

$\endgroup$

marked as duplicate by Stephan Kolassa, Michael Chernick, mkt, Ferdi, mdewey Jan 21 at 13:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 4
    $\begingroup$ "because the imbalance is not extreme I think it is possible to improve" - whether your data are imbalanced or not has no bearing on whether you can improve on a simple model. You can't improve on 50% accuracy for predicting a coin toss, no matter how much (balanced) data you have. You can't improve on 16.7% accuracy for predicting a die roll, no matter how much (imbalanced) data you have. $\endgroup$ – Stephan Kolassa Jan 20 at 17:19
  • $\begingroup$ Thanks for the link and comment Stephan. My point was that it's not like the null accuracy is 99.9%. Since you mentioned prediction of completely random events, are you implying that that the response variable is almost random with the current set of features? $\endgroup$ – Ali Jan 20 at 18:18
  • 2
    $\begingroup$ What leads you to believe that your features are strongly predictive of your outcome? $\endgroup$ – Sycorax Jan 20 at 18:49
  • 1
    $\begingroup$ Anyhow, accuracy is not a proper score function. See stats.stackexchange.com/questions/359909/… $\endgroup$ – kjetil b halvorsen Jan 20 at 20:08
  • $\begingroup$ Ali, what @stephen and sycorax are asking is why do you think you can do better? Are you sure the variables are suitably discrimatory ( no one else can decide that for you) $\endgroup$ – seanv507 Jan 20 at 20:27