14
$\begingroup$

I'm running GLMs and GAMs and I can't find a clear answer about if I should be balancing my data or not.

I'm trying to get useful descriptive models, not predictive models. So I haven't split my data into training and test. Two reasons, I didn't think that was necessary (since it isn't a predictive model) and I don't have a huge amount of data to begin with (200 samples for males, 100 samples for females). I'm investigating what factors (10) impact mortality after heart failure.

For some factors I have a massive imbalance. For example, I've separated the data on sex and 97% of females are non-smokers.

Would be it best to just omit this variable or do I attempt some balancing techniques? Do I leave it alone since it may just be representative of the real-world? From what I've read, balancing data means the model may no longer reflect reality. But in the case of the female smokers, it seems I just do not have enough data to see whether smoking impacts mortality after heart failure in females.

I wanted to apply a confusion matrix but I'm not predicting, so it doesn't seem applicable.

$\endgroup$

2 Answers 2

18
$\begingroup$

+1 to usεr11852's answer.

Don't worry about "unbalanced" data, as long as you use appropriate models. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? GLM/GAM are appropriate.

I didn't think that was necessary (since it isn't a predictive model)

Over-/undersampling does not make sense, whether you are using it in a predictive or inferential model.

I don't have a huge amount of data to begin with (200 samples for males, 100 samples for females). I'm investigating what factors (10) impact mortality after heart failure.

Assessing 10 factors with a sample size of only 300 is dubious, especially since you apparently also include interactions, like sex by smoking status. Consider using a more parsimonious model.

But in the case of the female smokers, it seems I just do not have enough data to see whether smoking impacts mortality after heart failure in females.

That is quite possible. Low information content, like having only three female smokers, will mean that your parameter estimates will be uncertain. That is unfortunately just a fact of your data. Oversampling these individuals will pretend to the model that you have more data than you really do, and parameter estimates will be more "certain" after oversampling - but of course you do not really have more data, you are just counting some observations multiple times. The only way to address this issue is to collect more data.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer. I'm just starting out with the 10 factors and then whittling them down to the most relevant ones, I hope that's making a more parsimonious model $\endgroup$
    – Nonya
    Jul 13, 2021 at 15:07
  • 1
    $\begingroup$ Hm. If you are running an inferential model, then I hope you know that stepwise predictor selection (or any other "data-driven" model selection) will invalidate all p values. $\endgroup$ Jul 13, 2021 at 15:10
  • $\begingroup$ I was using stepAIC and also pseudo R2 to compare, but this is a useful comment thank you. I am getting a bit lost in the ways to evaluate and compare logistic regression models $\endgroup$
    – Nonya
    Jul 13, 2021 at 15:15
  • 3
    $\begingroup$ Stepwise predictor selection is problematic for inference, no matter whether you use p values or AIC to decide between models. Be careful here. Good luck! $\endgroup$ Jul 13, 2021 at 15:17
  • $\begingroup$ I'll look into this more! $\endgroup$
    – Nonya
    Jul 13, 2021 at 15:19
17
$\begingroup$

I would strongly suggest to avoid rebalancing your data at first instance.

Especially when using a GLM/GAM to describe the tendencies of the sample at hand it makes little sense to try to up- or down-sample the original sample as this would immediately change the base-rates. GLM/GAM predictions are generally "well-calibrated" (i.e. there are accurate as probabilities estimated by the model) and resampling with destroy the intercept ($\beta_0$) that anchor our estimates.

We might want to use some re-sampling techniques (e.g. bootstrapping) to better inform our audience for aspects of sampling variance and finite-sample precision but that's a different point. Similarly if we care about predictive performance more, using regularisation (e.g. a ridge regression model) might be preferable, but again not a necessity for a "descriptive model"). Please note that, yes, it is perfectly reasonable to be worried about certain covariates having small absolute counts but that uncertainty will be captured by the resampling and/or the standard error associated with that coefficient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.