2
$\begingroup$

I know there's a minimum sample size calculation for logistic regression and I've found many discussions about it. However, what I'd like to know is what happens if the sample size is too high. Hypothetically, let's say for about 200 predictors, if I have 12,000 records, will there be any problems with the analysis? Should I check for something before I choose the data? I don't have actual proportion of the data but assuming 50% positive cases, will there be any problem with the assumptions and analysis?

$\endgroup$
  • $\begingroup$ "I don't have actual proportion of the data but assuming 50% will there be any problem with the assumptions and analysis?" 50% of what? $\endgroup$ – Sycorax Apr 3 '18 at 3:25
  • $\begingroup$ I didn't know it mattered. I'd say "positive cases". The outcome could be deceased or alive. $\endgroup$ – user6401955 Apr 3 '18 at 3:34
  • 1
    $\begingroup$ Perhaps look at stats.stackexchange.com/q/2516/601 $\endgroup$ – John Apr 3 '18 at 3:37
  • $\begingroup$ @John, it was a good read. Thank you for sharing. I think I'm even more confused now. But that aside, the discussion seems to be for linear regression. Can this be generalized to logistic? After all, they're both linear. Am I correct? Although, what confuses me is that in linear regression we are looking for the difference in mean, for instance. What's the effect size in logistic regression? How would you explain it? $\endgroup$ – user6401955 Apr 3 '18 at 14:36
1
$\begingroup$

I would argue that with 200 predictors that 12,000 records is small, maybe too small. Consider how many two factor models you have just assumed into existence. Unless you have the sincere belief that all 200 factors are the true model in nature, just the pairwise combinations are 19,900. You have vastly overrun your records.

If you cannot reduce that down to a reasonable model size using logic then you will have no ability to perform model selection to determine which of the models is a good model of nature. Mechanically, of course, there is no strict objection to having a model of 200 factors, but if everything is highly correlated then you probably do not really have 12,000 independent observations.

I would look closely at the relationships among my 200 variables before I went any further.

EDIT

The phenomenon you are looking at is called the Jeffreys-Lindley paradox and it is a theorem that states that if you have data drawn from a normal distribution and a sharp null hypothesis such as $\mu=k$ then all true null hypotheses will be falsified if the sample size is large enough. Note that this assumes that the computers have infinitely many digits and you can purchase $\pi$ or $e$ ounces of wheat.

This led to the discussion of small departures from normality, which led to the discussion of other distributions. There is a discussion out there that you can avoid this with a Bayesian solution. You cannot. This is an intrinsic problem with sharp null hypotheses. Bayesian methods, because they operate in the parameter space rather than the sample space, cannot process sharp null hypotheses because if the data is truly normally distributed then $\Pr(\mu=k)=0$ since it is a single point on a continuum and has zero measure. Bayes will not help you escape this problem if this is your true hypothesis and not one of convenience.

You will not face this problem, however, as you are already planning on taking a Bayesian way out. If you chose step-wise regression the model selection criterion; the AIC, BIC, DIC or whatever; allow you to escape the paradox.

When you use information criterion you are using stylized Bayesian procedures that are point estimates of the posterior density over the model space. If you have a variable that matters that is also very close to zero, this will be accounted for properly by the information criterion.

If you drop a variable, let us say $x_{153}$ from the regression, this is no different from setting $\beta_{153}=0,$ except that it bypasses the paradox entirely because Bayesian methods are not subject to the paradox because they operate in the parameter space.

So, the answer is that yes the paradox applies to logistic regression, but does not apply in your case.

Please pre-filter your predictors to strip out those that you believe are largely identical. Two hundred predictors are a lot of predictors for 12,000 records if you have no idea what the true model may be. This will be even more so if you believe that the 12000 are not truly independent as is often the case in life sciences and social sciences data.

I think you have the opposite problem.

$\endgroup$
  • $\begingroup$ I disagree with Dave. Many ML methods are perfectly suitable for this problem. Try out LASSO, logistic regression, or ridge regression for a start. $\endgroup$ – Oliver Angelil Apr 3 '18 at 10:45
  • $\begingroup$ @OliverAngelil, I agree with you. The discussion came about when we started talking how to discuss regular logistic regression with other ML models like you suggested. I questioned whether its okay to compare logistic done in statistical method to other machine learning models. If we use large datasets, the ml will work fine, but will it still be true for the statistical model? $\endgroup$ – user6401955 Apr 3 '18 at 13:03
  • $\begingroup$ @DaveHarris, i think you also make a good point. Assuming, application of stepwise method for feature selection, the features were reduced to, say, 14. How would this affect then? Can we still confidently say the results obtained from the analysis is reasonable? What I understood about sample size is, large datasets may not always be good thing. It may find even a small differences in effect size and become significant in linear regression (?). I don't know how this translates to logistic regression. $\endgroup$ – user6401955 Apr 3 '18 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.