1
$\begingroup$

I'm using MATLAB R2016a for binary classification (time series prediction) of a financial case. I have a good total accuracy (70~75%) but specificity is about 90% and sensitivity is about 60% and vice versa. Currently I'm using grid search to optimize my classification model (SVM, neural network, etc.) based on total accuracy. My data-set has balanced samples of binary output.

How can I improve results in this case? Can I use any other performance metric to take into account unbalanced sensitivity and specificity (as mentioned, in some cases I have higher specificity than sensitivity and in some cases vice versa.)?

$\endgroup$
  • $\begingroup$ I'm not sure how to do this in Matlab, but in R I favor maximizing Kappa over Accuracy, especially when class frequencies are unbalanced. $\endgroup$ – HEITZ May 18 '16 at 23:36
1
$\begingroup$

It seems like you may have answered your own question. You optimized your methods for proportion classified correctly, and so that's what came out nicely, possibly at the expense of other metrics. If you want, e.g., to maximize accuracy under the constraint that that both specificity and sensitivity are at least 65%, then try tuning the models and setting thresholds accordingly. If you care about false positives and misses equally, and the base rate is 50-50 too, then maximizing accuracy, as you've already done, seems like the way to go. But if one is worse than the other, try defining an appropriate loss function and try to maximize that instead.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.