2
$\begingroup$

Suppose I have a dataset where 100 patients have the disease (e.g. information such as height, smoking, weight, age, disease status) and 10000 patients do not have the disease (i.e. class imbalance).

I am interested in using Logistic Regression to try and understand what patient characteristics appear to influence the odds of having the disease or not. As such, there are significantly more patients without the disease compared to those who do not.

I fear that fitting a Logistic Regression on the entire dataset might partly invalidate the results as patients without the disease will have more influence in the model estimates. To potentially mitigate this problem, I am thinking of using Propensity Score Matching to select 100 patients who do not have the disease - in a way such that we only select patients without the disease so that they have an "approximate analog" in the disease set. As a result, I will have a dataset with only 200 patients and the the ratio of disease to non-disease will be balanced.

I had the following question: By using this Propensity Score Matching approach, I will end up discarding lots of information corresponding to the non-diseased patients and a result might be forfeiting large amounts of valuable information that might be beneficial to the model. However, by including this information, I fear that I risk "flooding" the model with too much information corresponding to the "non-diseased patients" and suppressing information belonging to the diseased patients.

In general - can Propensity Score Matching be used to mitigate problems/biases associated with class imbalance when fitting regression models to such types of problems?

Notes:

Improve this question
$\endgroup$
9
  • 4
    $\begingroup$ If the ratio of diseased to healthy patients reflects the truth, what are you afraid of? $\endgroup$
    – Dave
    Apr 16 at 20:39
  • $\begingroup$ @ Dave: Thank you for your reply! I watched a documentary on AI in which some researchers trained a Convolution Neural Network (CNN) to detect pictures of wolves vs dogs. A large number of the wolf pictures that the CNN was exposed to contained snowy white backgrounds - however, this was not true about the dog pictures. The CNN model performed successfully in the end, but when the researchers further studied the CNN, they found that the CNN was effectively a "snow classifier" and was not even looking at wolf characteristics, but rather if there were "large white patches" in the picture $\endgroup$
    – stats_noob
    Apr 16 at 20:42
  • 3
    $\begingroup$ I’ve heard that story before and am not convinced that it is a problem. If wolves tend to be in the snow more than dogs do, a snowy background really is a signal that the animal is a wolf and not a dog. $\endgroup$
    – Dave
    Apr 16 at 20:49
  • 2
    $\begingroup$ @stats_noob your sample will no longer reflect the distribution of patterns from each class that you will see in operational conditions. That is likely to introduce bias. $\endgroup$
    – Dikran Marsupial
    Apr 18 at 7:29
  • 2
    $\begingroup$ I just want to add that, to put it simplistically, matching is used to create patient cohorts that are identical in their prognostic factors such that they are essentially randomized with respect to some exposure, such that you can compare their outcomes to get an estimate of the effect of that exposure on the outcome. Here, you don't have an exposure as far as I can tell and "propensity of disease" matching will simply select patient groups where these patient factors like age, height are uncorrelated with disease precisely due to matching. That's the exact opposite of what you want. $\endgroup$
    – elbord77
    Apr 18 at 22:11

1 Answer 1

Reset to default
6
+50
$\begingroup$

Putting aside whether reducing class imbalance is even a good thing, propensity score matching or any other matching method would be a terrible way to reduce class imbalance. I presume your strategy would be to find 100 non-diseased cases that are similar on all your covariates (i.e., on the propensity score) to the diseased cases. What you will be left with is a group of diseased and a group of non-diseased patients who are identical to each other, meaning you can't predict which is which from the covariates. The goal of matching is to make it so that covariates don't predict the treatment variable (in this case, disease status), but the entire point of your analysis is to be able to predict disease status. So creating groups that are balanced on the covariates is a terrible idea, essentially ruining your study. Do not do this.

You may feel like propensity score matching sounds like a good method because its goal is to reduce imbalance in the covariates, but covariate imbalance is a completely different concept from class imbalance. Covariate imbalance concerns the association between treatment and the covariates (i.e., the very thing you are trying to study), and class imbalance concerns the sample size of the classes to be predicted. While it's true that matching would eliminate class imbalance by discarding a huge number of your non-diseased cases, it would also make it so that disease status cannot be predicted from the covariates. I reiterate, do not do this.

Improve this answer
$\endgroup$
2
  • $\begingroup$ @ Noah: Thank you for your answer! What kind of problems is Propensity Score Matching useful at solving? I heard that "selection bias" can be an example of such a problem? $\endgroup$
    – stats_noob
    Apr 19 at 1:09
  • 2
    $\begingroup$ "Selection bias" is an ambiguous term with multiple meanings, so I don't use it. PSM is used for adjusting for confounding by measured variables in an observational study that seeks to estimate the causal effect of a treatment on an outcome. It creates "comparable" groups that are (ideally) identical except for the treatment, so the comparison of the outcomes between the treatment groups cannot be said to be due to anything but the treatment. $\endgroup$
    – Noah
    Apr 19 at 4:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.