Determine baseline for a machine learning project

Question

I work with a unbalanced data set (it is about people who actually bought stuff):

Bought stuff: Yes  ~ 3%
Bought stuff: NO   ~97%

The most important task for my machine learning model, is to optimize the sensitivity (I want to "catch" all the "Yes" people, the 3%).

But I was wondering how I could define the baseline. I read this article (https://machinelearningmastery.com/how-to-get-baseline-results-and-why-they-matter/) where is written: "Classification: select the class that has the most observarions and use that class as the result for all predictions".

But, because sensitivity is the most important, can I say that my baseline is 3% (the Yes class, because when you randomly guess.. you will guess statistically 3 people as buyers from the 100).

Answer 1

The most important task for my machine learning model, is to optimize the sensitivity (I want to "catch" all the "Yes" people, the 3%).

Taking this sentence literally, the baseline (guess "Yes" for everybody) is the best possible method - this will get 100% sensitivity and there is no way to improve. Obviously you want to also get good specificity without compromising sensitivity but how much of a compromise is still worth it? (e.g. will you be willing to reduce sensitivity to 95% to get 100% specificity?) There is no single good answer, it really depends on your case and sensitivity alone is impossible to interpret.

Your question IMHO illustrates a wider problem with thinking in terms of sensitivity and specificity. I would suggest that you define a cost function - what is the cost/utility of true positives, true negatives, false positives and false negatives (those all can have dramatically different costs!) and then try to find a classifier that minimizes expected cost (maximizes expected utility).

You can then compare which of the baseline classifiers (in your case just giving the same answer for all inputs) has lower expected cost and use this one.

Frank Harrell has some more thoughts on this topic: http://www.fharrell.com/post/classification/

Answer 2

Generally, to build a baseline, I take a look what my data is - specially the fraction of classes, how many classes there are and finally how many features. In your case, saying "Yes" to everything increases the sensitivity but I am assuming that's not what you want to do, for obvious reasons.

I will share my rule of thumb for building baseline in your case (rather what I know about your case):

1. Its a binary classification problem: First thing that clicks me is SVMs - as it has been said by others - SVMs gives you an optimal solution whereas most other approaches will give you a good enough solution. In machine learning terms SVMs have low variance.
2. Since, your classes are way imbalanced, I would suggest using appropriate class weights - for SVMs or whatever classifiers you choose.
3. I am assuming you have a lot of features. If you don't, ignore this step. If you do, reduce these features by random forest feature ranking or some other supervised feature reduction techniques. I would advise PCA since its unsupervised and you may end-up getting undesirable results because high class imbalance.