2
$\begingroup$

I work with a unbalanced data set (it is about people who actually bought stuff):

Bought stuff: Yes  ~ 3%
Bought stuff: NO   ~97%

The most important task for my machine learning model, is to optimize the sensitivity (I want to "catch" all the "Yes" people, the 3%).

But I was wondering how I could define the baseline. I read this article (https://machinelearningmastery.com/how-to-get-baseline-results-and-why-they-matter/) where is written: "Classification: select the class that has the most observarions and use that class as the result for all predictions".

But, because sensitivity is the most important, can I say that my baseline is 3% (the Yes class, because when you randomly guess.. you will guess statistically 3 people as buyers from the 100).

$\endgroup$
2
$\begingroup$

The most important task for my machine learning model, is to optimize the sensitivity (I want to "catch" all the "Yes" people, the 3%).

Taking this sentence literally, the baseline (guess "Yes" for everybody) is the best possible method - this will get 100% sensitivity and there is no way to improve. Obviously you want to also get good specificity without compromising sensitivity but how much of a compromise is still worth it? (e.g. will you be willing to reduce sensitivity to 95% to get 100% specificity?) There is no single good answer, it really depends on your case and sensitivity alone is impossible to interpret.

Your question IMHO illustrates a wider problem with thinking in terms of sensitivity and specificity. I would suggest that you define a cost function - what is the cost/utility of true positives, true negatives, false positives and false negatives (those all can have dramatically different costs!) and then try to find a classifier that minimizes expected cost (maximizes expected utility).

You can then compare which of the baseline classifiers (in your case just giving the same answer for all inputs) has lower expected cost and use this one.

Frank Harrell has some more thoughts on this topic: http://www.fharrell.com/post/classification/

$\endgroup$
  • $\begingroup$ Yes, you are right: there must be a balance between Sensitivity and Specificity. But, It is in my task the most important to catch all the "Yes" people (these are the ones that add value to the business). I will check your suggestion (with the cost function), but is it impossible to say: alright, seen the problem: your baseline is 3% (because only 3 of the 100 people belong to the Yes class). $\endgroup$ – R overflow Mar 20 '18 at 16:16
  • $\begingroup$ I think you are confusing two meanings of the word "baseline". It may mean a) a simpler/naive procedure to solve the problem or b) the result of applying such a method. Responding always "Yes" leads to 100% sensitivity and 3% accuracy (see en.wikipedia.org/wiki/Sensitivity_and_specificity for definitions). Improving over this baseline in sensitivity alone is impossible, improving in accuracy alone is trivial (always "No" has accuracy of 97%). Unless you state what tradeoff between sensitivity and accuracy/specificity are you willing to make, the baseline is of little value. $\endgroup$ – Martin Modrák Mar 21 '18 at 6:46
0
$\begingroup$

Generally, to build a baseline, I take a look what my data is - specially the fraction of classes, how many classes there are and finally how many features. In your case, saying "Yes" to everything increases the sensitivity but I am assuming that's not what you want to do, for obvious reasons.

I will share my rule of thumb for building baseline in your case (rather what I know about your case):

  1. Its a binary classification problem: First thing that clicks me is SVMs - as it has been said by others - SVMs gives you an optimal solution whereas most other approaches will give you a good enough solution. In machine learning terms SVMs have low variance.
  2. Since, your classes are way imbalanced, I would suggest using appropriate class weights - for SVMs or whatever classifiers you choose.
  3. I am assuming you have a lot of features. If you don't, ignore this step. If you do, reduce these features by random forest feature ranking or some other supervised feature reduction techniques. I would advise PCA since its unsupervised and you may end-up getting undesirable results because high class imbalance.
$\endgroup$
  • $\begingroup$ I used a Random Forest, because that was the classifier with the best performance (Sensitivity and Specificity), But as said before: especially the Sensitivity was important (of course, I know that there must be a balance). But my question is: how can I see what the baseline is? (A binary classification task, with highly imbalanced classes). Thanks! $\endgroup$ – R overflow Mar 20 '18 at 16:12
  • $\begingroup$ what do you mean by "how can I see" $\endgroup$ – silent_dev Mar 21 '18 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.