6
$\begingroup$

I read online that 1:10 rule is based on the frequency of lower occurring class.

I have a dataset with 4712 records. There are 1558 records labeled yes, and 3554 records labeled no. In my case, the frequency of the less commonly occurring class is 1558. According to the 1:10 rule, am I right to understand that it is calculated like 1558/10 = 155.8 further equals 150 predictors? So In my logistic regression, I can use 150 variables/input features to the model without the risk of overfitting. Am I right?

  1. Do we also have to look at the frequency of the other (high occurring) class to determine the no of predictors that I can use? If yes, can you share me as to what has to be done to determine the predictor count?

(I am aware that we could also use 1:20 or 1:50 rule. But my question is mainly:)

  1. Are there any other considerations for determining the number of predictors in a logistic regression model?

  2. How do people calculate the minimum sample size required for analysis based on this? Is events per variable about that? (For example, I can't have 20 records and 60 predictors. It may not make sense.)

Similarly, can you people help me understand how to determine the number of predictors and sample size required for logistic regression? Is it called a 'data hungry model'?

$\endgroup$
3

1 Answer 1

8
$\begingroup$

This is a "rule of thumb" (English usage: a rough guide, in this case a guide to study design and initial modeling), not a strict "rule" that by itself ensures a lack of overfitting. Harrell's course notes and book provide (in their 4th Chapters) references to several studies that evaluated the sample sizes necessary to avoid overfitting in low signal-to-noise situations like observational studies in medicine or social sciences. Values on the order of 10 to 20 members of the minority class (or events in survival analysis) per candidate predictor (hence the generic "events-per-predictor" terminology) were found to do a reasonable job of avoiding overfitting. You can think about this as the minority class being the hardest to fit if there are too few cases, and the harder you try to fit (by adding in more predictors) the more likely that you will overfit.

For your questions:

  1. I can't rule out some influence of the majority-class size, but as this is a rough guide, not a strict "rule," I don't see much reason to worry about the majority-class size.

  2. As the Harrell references linked above make clear, much also depends on the nature of the study. Well-controlled experiments or experiments with small errors in the physical sciences might only need a lower ratio. If the values of the predictor variables are narrowly distributed or you have small numbers of cases with a particular value of a categorical predictor, you might need a higher ratio. If you need a precise estimate of the intercept in a logistic regression you may need more. As noted in comments on your question, there are alternatives to the simple events-per-predictor guideline that you could consider.

  3. Events-per-predictor is a starting point for study design and analysis in the fields for which this rule of thumb was determined. You still need to document that your model did not, in fact, overfit substantially. Building models from multiple bootstrapped samples, evaluated against the full data set, is one good way to do this. Harrell's rms package in R provides tools for such validation and calibration of many types of regression models, including logistic. If you are planning a prospective study and have some ideas about the spreads of your predictor values and their associations with outcome, you can do a formal power analysis to help determine the necessary study size.

Note that you can have "20 records and 60 predictors" or even more dramatic excesses of predictors to cases in some areas of interest, which can be called the "$p \gg n$" problem. For example, studies of gene expression in biology can have almost 20,000 potential predictors (the mRNA level for each of 20,000 genes) to associate with only a few dozen events (e.g., cancer deaths). Minimizing overfitting in such cases often involves a regularization approach like LASSO or ridge regression, which puts a penalty on the magnitudes of regression coefficients (many or most penalized down to 0 in LASSO). This trades off some systematic bias in the predictions against less variance when the model is applied to new data. These methods can be thought of as reducing the effective numbers of predictors as their coefficient magnitudes are reduced.

As you seem to be just starting serious modeling of this type, I strongly recommend that you consult references like those by Harrell linked above to understand the principles, and find a local statistics expert who can work with you and your colleagues to apply the principles to your study.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.