Apply logistic regression to binary data - Need guidance

Question

Good morning.

I am trying to predict the probability of death of people suffering from a disease, based on their age and gender.

Currently from the data (approximately 50,000 people), 43699 survive and 1035 die, here I see the first problem, I think there is an unbalanced distribution and I do not know how to treat it.

Let's continue...

First, graph the people who died and those who did not, in a graph of age, gender (muscle = 1, female = 0). As you can see in the first image, there is too much data for a good visualization.

Even taking out the survivors, you can see that the deceased are distributed in both genders and in almost all ages.

Try to use the sklearn library to apply a logistic regression as I think it is the most suitable for this purpose. If I manage to run it, I get other problems.

First, the frontier of decision, it doesn't give me any information (I think, from my humble knowledge, that this happens because there is a distribution of the deceased in almost all ages and genders).

Second, the accuracy of the model is too bad, I attach the data:

>>> print(classification_report(prueba, predictions))
              precision    recall  f1-score   support

           0       0.98      1.00      0.99     43699
           1       0.34      0.01      0.02      1035

    accuracy                           0.98     44734
   macro avg       0.66      0.51      0.50     44734
weighted avg       0.96      0.98      0.97     44734

Where: 0 represents survivor, 1 represents death

After this long introduction, I wanted to ask you:

1. Is there any way to correct this model for the present data?
2. If not, is there another model that is more useful to me? Maybe k-Means
3. Is there a way to graph it that will give me a better representation?

Maybe the input data does not contain a direct relation and you can't find any productive model.

Any help you can give me for this data is welcome.

Thank you!

Answer 1

You are correct to use logistic regression to predict death in your cohort. However before fitting a model to your data, you should extensively analyze your data:

1. Check for missing values. This is especially true for very large datasets (e.g. 50,000 individuals).

Currently from the data (approximately 50,000 people), 43699 survive and 1035 die, here I see the first problem, I think there is an unbalanced distribution and I do not know how to treat it.

2. Unbalanced distributions might be resolvable with sub-setting. You could match patients who progress with patients who don't progress based on age, sex, income, baseline characteristics, etc. Thereby you would create a smaller dataset with fewer observations but less skewed outcomes.

3. Explore your data. You have done that partially, but you treated gender as a continuous variable. Try converting it to a factor and use scatter and transparency to avoid overlapping of points. Alternatively you could plot colored histograms.

4. Fit your model. To give a good answer concerning which model to use and how to apply it, we need more information about your dataset and the code you have already run. Presumably logistic regression would be the right thing to do.