1
$\begingroup$

I've generated a gradient boosting model using r-caret on data which I expect to have little to no predictive value. Class distribution is heavily skewed with ~15000 negative and ~1000 positive. Caret's confusionMatrix function reports a sensitivity of ~0.02 and a specificity of >0.99. How should I interpret this?

$\endgroup$
1
$\begingroup$

Definitions:

Sensitivity (SEN) is defined as TP/(TP+FN). It represents the ratio of positive samples labeled correctly out of all the positive samples. Similarly, Specificity (SPC) is defined as TN/(TN+FP), which is the ratio of negative samples labeled correctly out of all negative samples.

Now, you have a 15:1 negative:positive samples ratio. A very naive algorithm, which always returns "False" regardless of the sample given, would naturally produce SPC=1 and SEN=0. A random algorithm, which returns "True" 1% of the time (regardless of the sample given) and "False" 99% of the time, would produce on average SEN=0.01 and SPC=0.99. This means that your model is pretty much in the range of a random model, and not very useful for classification.

Try oversampling (or better, get more positive data) to improve the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.