1
$\begingroup$

I am working on a classification problem, with unbalanced classes :

Number of positive examples: ~200k; Number of negative example is ~230 Millions examples.

The only two requirements that I have is: Using AUC for evaluation and the evaluation should be at natural rate i.e. 200k/230M = 0.0008.

My question here is, knowing that using all that data is quite impossible because of performance constraints (currently the limit is around 6M sample of the data), what would be the strategy to train, cv and test a model ?

Two propositions came up, we can't decide which one is the best practice in this case:

  • Train on 6M neg + 190k pos, Crossvalidation using natural rate, Test using natural rate
  • Train on 6M neg + 5k pos (natural rate), CV and Test are all in natural rate
Improve this question
$\endgroup$
5
  • $\begingroup$ Why those two requirements? $\endgroup$
    – user2974951
    Oct 19, 2020 at 7:38
  • $\begingroup$ @user2974951 They are from my client, personally I understand the evaluation on natural rate, but I disagree with the AUC, I'd rather use AUCPR or a form of FbetaScore. $\endgroup$
    – deltascience
    Oct 19, 2020 at 7:43
  • 1
    $\begingroup$ I can understand the client asking for AUC (probably what they are familiar with, although not a good reason), but why ask for a model to be trained with the same ratio as the data? That seems a little weird, have you inquired more about this point? Is the client expecting the same ratio on new data? Even so, also not a good reason. $\endgroup$
    – user2974951
    Oct 19, 2020 at 7:49
  • $\begingroup$ What is meant by "evaluation should be at natural rate"? That out of 10,000 samples, 80 should be classified as positive? $\endgroup$
    – Stephan Kolassa
    Oct 19, 2020 at 9:31
  • $\begingroup$ @StephanKolassa just changed my comment: evaluation should be done on a test & validation set that have nb_positive_examples/nb_negative_examples = 0.008, the reason behind that is to estimate our performance in production where only 0.008 of the entries would be positive. $\endgroup$
    – deltascience
    Oct 19, 2020 at 13:32

2 Answers 2

Reset to default
3
$\begingroup$

I disagree with user2974951's answer. You should aim at probabilistic classifications that are well-calibrated and sharp. Oversampling the minority class will bias your predictions, so don't do it. Use a representative sample of your initial data for training your model.

Once you have your probabilistic classifications, you can calculate your AUC. In addition, you can tweak your cutoff threshold (which I am not too keen on) based on your training sample, until you get a "positive" hard classification rate of 0.008, as the client requires. (Which doesn't make much sense to me, per the same reasoning.)

Improve this answer
$\endgroup$
9
  • $\begingroup$ Where does oversampling come into play? OP has 200k positive cases in total, the first option would use 190k cases? $\endgroup$
    – user2974951
    Oct 19, 2020 at 10:24
  • 1
    $\begingroup$ @user2974951: taking 190k positive cases would sample the positive cases at a far higher rate than they are present in the full sample. That is oversampling, and the model will be led to believe the positive cases are far more prevalent than they really are. $\endgroup$
    – Stephan Kolassa
    Oct 19, 2020 at 10:32
  • $\begingroup$ @StephanKolassa thanks for the answer, could you explain what you mean by Which doesn't make much sense to me, per the same reasoning.) Are you referring to the 0.008 rate ? what would you suggest then, knowing that any random sample from the data will give you similar proportions pos/neg ? $\endgroup$
    – deltascience
    Oct 19, 2020 at 13:02
  • $\begingroup$ Requiring a fixed rate makes no sense. If you have 1000 instances, but only 4 make the predictive cutoff you have learned, then that should not be a reason to look askance at your model. It may simply be that your 1000 instances happened to have predictor values that are less predictive of a positive case. As long as your predictive probabilities are well-calibrated and sharp, i.e., are probabilistically correct, we should not worry how many of them exceed a threshold. The rate can only be reasonably looked at if the predictor distribution is the same between training and test. If that. $\endgroup$
    – Stephan Kolassa
    Oct 19, 2020 at 13:18
  • 1
    $\begingroup$ @StephanKolassa oh wait, just saw that I misunderstood your question in my question's comment. The natural rate requirement, doesn't mean that 0.008 have to be classified positive, I meant that in my test & validation set the proportion of pos/neg should be 0.008. $\endgroup$
    – deltascience
    Oct 19, 2020 at 13:29
0
$\begingroup$

If you must follow the requirements, then the first option would be better for the simple reason that you have more data for the minority class, so you should be able to capture more of the variability in this class. As opposed to the second option where you would first have to (find a way to) select a small subset of all the data (and throw the other data out?, this seems unnecessarily wasteful).

Edit: some more clarification. The first option would see you select a sample of your data (although much bigger than the second option) and the perform CV with folds, ensuring that each fold is trained with the same ratio of positives and negatives, as your original sample. In this way you would preserve the required "natural rate" while at the same time having more data available, so I don't see how this is a worse option than number 2.

I would, however, suggest a third option, one that would require no removing of data. Keep all your data as is, but then perform a generalization of CV, whereby you train your desired model multiple times with subsamples of your data, while keeping the ratio intact (stratified sampling). So each iteration you select a new subsample, getting new data into your training sample. This way, given enough repetitions, you should get most of your data through your model, and hence get a much better look at the performance of the model. The best cutoff for AUC would then be selected from the folds, where the natural rate was in effect.

This is similar to repeated CV, where you repeat CV multiple times, each time with new folds, so as to try and randomize the training and test data as much as possible, and get a more accurate result.

Improve this answer
$\endgroup$
7
  • $\begingroup$ This answer is confusing to me for several reasons. What is most unclear to me, I think, is why training on biased data will lead to a better or "not worse" model than training on a slightly smaller data set that is not biased. The "capturing more variability" argument seems quite vague. $\endgroup$
    – einar
    Oct 22, 2020 at 8:12
  • $\begingroup$ @einar In what way is the training data biased? When we subsample the data we retain the ratio of positives and negatives, the exact same as if you were to only subsample it once. The only difference is that here we would do this multiple times. $\endgroup$
    – user2974951
    Oct 22, 2020 at 8:25
  • $\begingroup$ Sorry I thought you must be talking about the usual approach some people take where you "oversample" the minority class in the training data but keep the test data at the correct ratio. Another confusion I have: CV is primarily a method for getting an error estimate, how do we get a more accurate model with it? $\endgroup$
    – einar
    Oct 22, 2020 at 8:37
  • 1
    $\begingroup$ @einar As I see it, the two options are very much related, the difference is that in option 2 we first select a subsample, which is now fixed, get 1 model, and use this model to predict the test, all the data which was left out is thrown in the trash (and we lose a lot of data from the minority class). With option 1, or with my suggestion, which is the same without any initial subsampling, we would subsample the data many times, say n times, get n subsamples and get n models. How we obtain one final model from this second approach... there are different ways. (cont.) $\endgroup$
    – user2974951
    Oct 22, 2020 at 8:57
  • 1
    $\begingroup$ One option would be to not do it, but rather keep the n models, get predictions from each one and then aggregate the predictions. Another option would be to build the final model on all the data, but use the knowledge obtained in the n separate models to calibrate the final model, to take into account the "natural frequencies". $\endgroup$
    – user2974951
    Oct 22, 2020 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.