2
$\begingroup$

I am using re-sampling methods to address the imbalance between classes for my binary classification problem.

I am not sure how to measure the performance of my model on the test set:

  • should I re-sample the test set to have an idea how my model is performing on the test set compared to the training set?
  • or should I measure the performance on the original test set and, to compare it with the training set, also measure the performance on the original training set?
Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Why do you think you need to resample? $\endgroup$
    – Matthew Drury
    Aug 25, 2017 at 16:30
  • $\begingroup$ I use resampling in order to give a higher weight to positive samples, which belong to the class that is the most important for me (misclassifying a positive sample is more costly from a business point of view than misclassifying a negative sample). I would think that my second bullet point is the best methodological approach, but my first bullet point still makes some sense IMO (from a performance point of view, not a business point of view). $\endgroup$
    – Tanguy
    Aug 28, 2017 at 8:43
  • 1
    $\begingroup$ Why not fit a probabilistic model? I.e. one that gives probabilities of class membership? $\endgroup$
    – Matthew Drury
    Aug 28, 2017 at 13:06
  • $\begingroup$ In thise case, I am using RF as they are performing pretty well. I believe RF can not be qualified as probabilistic models in classification, but I guess the probability of a class can be approximated by the voting proportion to each class. Could you please guide me in showing how this could relate to the re-sampling methodology in order to correctly measure the model's performance? $\endgroup$
    – Tanguy
    Aug 28, 2017 at 15:51
  • 1
    $\begingroup$ I can't really help you with resampling, because I don't understand it very well. I've been vocal about my suspicion that it is possibly snake oil. On RF, it is a probabilistic model, as each tree is probabilistic by averaging the labels in each terminal node. $\endgroup$
    – Matthew Drury
    Aug 28, 2017 at 16:58

1 Answer 1

Reset to default
1
$\begingroup$

The most realistic option would be to resample your training set however you see fit to train a model that can deal with the class imbalance and to leave the test set untouched to test if that model then can actually deal with imbalance. If you take the imbalance out of the test set, you will not see if your model can deal with it or not.

If you need to do a lot of model tuning and model choice, I would also do a CV within the training set, do the resamplings each time within the training folds and compare your models in the untouched other fold. Once you have chosen a model and it's parameters, you may retrain it in a resample training set that includes all the folds and (without changing the parameters of he model at this point) and obtain a performance measure from the test set.

Don't use accuracy to measure performance at any point in time since all your out of sample predictions will be made on imbalanced sets.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I am still unconvinced that resampling solves any actual problem. If you have a strong understanding of this technique, please offer an answer to my quesitons here: stats.stackexchange.com/questions/285231/…, stats.stackexchange.com/questions/247871/… $\endgroup$
    – Matthew Drury
    Aug 29, 2017 at 13:32
  • $\begingroup$ I was mostly answering his questions within the framework of resampling. Personally, I prefer changing the cutoff and if that doesn't help perhaps feature engineering. I didn't dismiss his premise though since for example in the book applied predictive modelling it is presented as one way to deal with imbalance. Within the resampling framework, the most important point was to not resample your test set or you will never know how you perform. $\endgroup$
    – David Ernst
    Aug 29, 2017 at 13:36
  • 1
    $\begingroup$ I know this is commonly presented in the literature, and in many text books, yet I cannot get a good, straight answer at exactly what it means to "deal with unbalanced data". I've routinely fir gradient boosted models (minimizing log loss) to 200/1 balanced data with no issues. Why is imbalance a thing that must be "dealt with"? $\endgroup$
    – Matthew Drury
    Aug 29, 2017 at 13:43
  • $\begingroup$ Because many algorithms will implicitly optimize accuracy which is inadequate with strong class and cost imbalance. If you have found an algorithm that doesn't do that for the data-sets you had, all the better. $\endgroup$
    – David Ernst
    Aug 29, 2017 at 13:59
  • 1
    $\begingroup$ Which algorithms are you thinking of? Most classifiers in common use optimize log-loss (trees, random forest, gradient boosting, neural networks, regression, additive models, etc). Of the well known algorithms, only SVM falls into that class, and this is probably one large reason it has fallen out of favor in the current day. To put a blunt point on it, if you are ever optimizing accuracy, you have a pretty large conceptual problem. $\endgroup$
    – Matthew Drury
    Aug 29, 2017 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.