0
$\begingroup$

I see a few discussions that suggest downsampling is never correct for logistic regression or suggesting that you have to do bias term corrections post-hoc:

The reasoning makes sense to me. However, I am training my logistic regression via SGD because of the amount of data I have (sparse features and a lot of samples). I am not able to get any meaningful convergence without downsampling.

If I primarily care about ordering and not about calibrated expectation of response variable is it okay to downsample in order for my SGD to converge?

Alternatively, I can set a very low learning rate and try to use all of my data but would rather avoid that if feasible.

$\endgroup$
  • $\begingroup$ how big is your data set? have you tried logistic regression code that uses sparse matrix implementations (eg glmnet). what logistic regression SGD code are you using? $\endgroup$ – seanv507 Dec 11 '18 at 23:22
1
$\begingroup$

Downsampling is never an appropriate statistical technique. And note that the $c$-index (concordance probability; AUROC) is unaffected by imbalance anyway. For more details see this.

$\endgroup$
  • $\begingroup$ Thanks for your reply. Do you recommend then just attempting to get the SGD to work on the entire dataset? $\endgroup$ – John Dec 11 '18 at 20:39
  • $\begingroup$ "Downsampling is never an appropriate statistical technique" what do you mean? In what context? Any sources? $\endgroup$ – user2974951 Dec 12 '18 at 7:32
  • $\begingroup$ Think first about sampling/data acquisition. Can you think of a valid statistical method that wants you to remove data after sampling when the data are not in error? Then consider that you are only considering this because you are using an improper accuracy scoring rule. The link I provided has details. Please read it. $\endgroup$ – Frank Harrell Dec 12 '18 at 13:32
0
$\begingroup$

If you're just trying to get good relative ordering just give it a try! Evaluate the performance on a held out test set (without downsampling), and see what kind of AUROC you get there.

If you're downsampling at random (just taking a small fraction of your dataset) then the only real downside is that you're not taking advantage of all your data.

If you're downsampling the majority class only, your model will no longer be well calibrated, but relative ordering should be preserved.

Another possibility is using mini-batch SGD, you should get faster convergence, and still use all available data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.