Logistic regression balanced dataset

Question

I am quite inexperienced using logistic regression and am having trouble understanding my data and how the regression behaves. Here's the outline of my problem:

I have a (medical) test that gives a yes/no result (basically like a pregnacy test) and I have a continuous variable that I want to use to predict the test result. Furthermore, I want to divide my data into two subsets (according to another attribute) and check whether the test gives different results based on subset membership. I ran a logistic regression on these two subsets respectively (using python's statsmodels package) to compare the output. Looking at the plots, the fitted lines for each subset are quite a bit apart (sorry this is not very specific).

The subsets are quite different from each other wrt the test results, i.e. subset 1 has six times more positive than negative results whereas for subset two it's only about 1.5 times more. I then manipulated the data of the two subsets in such a way that I have equal numbers of positive and negative test results (by excluding positive datapoints randomly) in each of them. When I now run the two regressions, the lines are basically the same.

I am wondering what this tells me or if I can draw any conclusions from it, i.e. "the test works the same given the continuous independent variable, irrespective of subset membership". But this is only true when I balance the datasets and the output of the regression is clearly different with the original data. Basically, I am wondering if this difference is due to different distributions of the continuous variable in the two subsets (which are different, i.e. they have different shapes but means and medians are relatively close to each other) or if the test actually does not only work based on what the continuous variable represents but is affected by something else as well (which I think is really unlikely).

I hope I'm somewhat making sense. Maybe someone has an alternative suggestion of what to do to investigate this question and some insight into what could explain such a behaviour or where I'm making a mistake in my thought process.

Thanks in advance!

Answer 1

A few things:

• Do not split your data. You should be using the attribute you're using to split the data as a variable in the model.

• Do not manipulate the outcome. Analyze the data as they are.

• I would not attempt to interpret the results of your procedure as the frequency properties of the procedure are not well understood. If you can post your data, or a reasonable facsimile, we can show you an appropriate method of analyzes.