0
$\begingroup$

I see a few discussions that suggest downsampling is never correct for logistic regression or suggesting that you have to do bias term corrections post-hoc:

The reasoning makes sense to me. However, I am training my logistic regression via SGD because of the amount of data I have (sparse features and a lot of samples). I am not able to get any meaningful convergence without downsampling.

If I primarily care about ordering and not about calibrated expectation of response variable is it okay to downsample in order for my SGD to converge?

Alternatively, I can set a very low learning rate and try to use all of my data but would rather avoid that if feasible.

Improve this question
$\endgroup$
1
  • $\begingroup$ how big is your data set? have you tried logistic regression code that uses sparse matrix implementations (eg glmnet). what logistic regression SGD code are you using? $\endgroup$
    – seanv507
    Dec 11, 2018 at 23:22

2 Answers 2

Reset to default
1
$\begingroup$

Downsampling is never an appropriate statistical technique. And note that the $c$-index (concordance probability; AUROC) is unaffected by imbalance anyway. For more details see this.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your reply. Do you recommend then just attempting to get the SGD to work on the entire dataset? $\endgroup$
    – John
    Dec 11, 2018 at 20:39
  • $\begingroup$ "Downsampling is never an appropriate statistical technique" what do you mean? In what context? Any sources? $\endgroup$
    – user2974951
    Dec 12, 2018 at 7:32
  • $\begingroup$ Think first about sampling/data acquisition. Can you think of a valid statistical method that wants you to remove data after sampling when the data are not in error? Then consider that you are only considering this because you are using an improper accuracy scoring rule. The link I provided has details. Please read it. $\endgroup$
    – Frank Harrell
    Dec 12, 2018 at 13:32
0
$\begingroup$

If you're just trying to get good relative ordering just give it a try! Evaluate the performance on a held out test set (without downsampling), and see what kind of AUROC you get there.

If you're downsampling at random (just taking a small fraction of your dataset) then the only real downside is that you're not taking advantage of all your data.

If you're downsampling the majority class only, your model will no longer be well calibrated, but relative ordering should be preserved.

Another possibility is using mini-batch SGD, you should get faster convergence, and still use all available data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.