2
$\begingroup$

I'm looking for clarification on how many fixed effects parameters can be reasonably included in a mixed logistic regression without saturating or over-fitting the model. As a background, my dataset consists of roughly 40,000 telemetry locations from 20 individual caribou (~2,000 locations per individual), and I'm using a mixed logistic regression framework to compare the habitat conditions at used telemetry locations (1's) to randomly sampled 'available' locations (0's). I specify the individual as a random effect to account for the nested data structure, and my predictor variables are things like elevation, land cover, terrain ruggedness, distance to roads and water, etc.

I'm familiar with the convention suggesting a minimum of 10 observations per parameter for linear and logistic regressions WITHOUT random effects (referenced in http://www.sciencedirect.com/science/article/pii/S0895435615000141 and https://academic.oup.com/aje/article/165/6/710/63906/Relaxing-the-Rule-of-Ten-Events-per-Variable-in), but in my situation I'm confused about whether to treat individuals or telemetry locations as my 'observational unit'. There are some useful links in this similar question: Is there a general rule about max nr of variables to use in (generalized) linear model?, and in several other questions pertaining to sample size for linear and logistic models, but it is not clear whether or how the rules of thumb discussed apply to a nested situation.

This is an important consideration because depending on how I calculate my sample size (i.e. based on total telemetry locations, telemetry locations per individual, or number of individuals), the maximum number of parameters advisable to include in a mixed model would vary drastically according to the 10:1 rule of thumb, from 4,000 (40,000 locations/10) to 20 (2,000 locations per individual/10) to 2 (20 individuals/10). For reference, I would ideally include ~15 candidate variables in a full model. If I could only include 2 variables without overfitting the model, I fear it would be a very poor model indeed. In the literature in my field, similar situations reference the individual animal as the experimental unit, however overfitting is rarely mentioned and the number of parameters included in models is highly variable.

My questions boil down to these:

1. What is the most appropriate way to calculate sample size in a nested study design such as mine? The total number of locations, the number of locations per individual, or the number of individuals?

2. Which measure of sample size should be used in identifying the maximum number of parameters to include in a mixed model to avoid overfitting the model in a nested study design such as mine? Does the 10:1 rule of thumb still work in a mixed modeling framework?

Improve this question
$\endgroup$
2
  • $\begingroup$ just a suggestion to add some information to your question: to what level of nesting do the candidate variables apply? $\endgroup$
    – IWS
    Commented Jan 17, 2017 at 15:09
  • $\begingroup$ Thanks @IWS, good point. Individuals are sampled from the same landscape and temporal period and all predictor variables pertain to that landscape and timeframe, thus I wouldn't ordinarily consider any nesting structure in the candidate predictor variables. $\endgroup$
    – dmacnearney
    Commented Jan 17, 2017 at 16:00

2 Answers 2

Reset to default
1
$\begingroup$

I wouldn't worry so much about rules of thumb as the number of data points that will be brought to bare for the parameters you are keen to have estimates for, as well as the variances of those estimates. For example, it sounds like you will have ample N to estimate the slopes of elevation, land cover, etc... (2000 per caribou, and 40k overall with slightly adjusted intercepts per caribou). But your estimate of the global intercept will likely be pretty noisy, since you only have 20 caribou. Although this is ultimately an empirical question—–it's possible that you have little variance across caribou. Have you measured the intraclass correlation coefficient to determine the necessity of a mixed effects model?

In general, it's hard to answer your question without knowing the model structure. Will you have random slopes for some predictors? What is the goal of the model? If it is merely a predictive model, than cross-validation or stepwise model selection techniques can give you an empirical answer to your question regardless of any rules of thumb. If some models fail to converge then there's your answer.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the insight @ericgtaylor, I hadn't considered that it might be a moot point if final models pass a cross validation test. As for the necessity of a mixed model, Ive been following onlinelibrary.wiley.com/doi/10.1111/j.1365-2656.2006.01106.x/… which advocates for mixed models as way to account for unbalanced sample sizes between individuals in addition to improving model performance through estimation of random slopes and intercepts. I will look into the ICC for interest's sake, but given the data structure I don't think I'll move away from the mixed model framework. $\endgroup$
    – dmacnearney
    Commented Aug 1, 2017 at 16:04
  • $\begingroup$ I had another thought, are cross-validation results reliable when working with an oversaturated model? Might this situation lead to a false-positive interpretation? $\endgroup$
    – dmacnearney
    Commented Aug 1, 2017 at 17:43
  • $\begingroup$ Yea, I get ya. I also prefer the mixed models framework as a default with nested data, regardless of the ICC. Re cross-validation with oversaturated models, the CV results should be very poor if the model is truly oversaturated. The point of CV is just to assess the variability in model performance on a "hold-out" set, when training the model on different partitions of the data. If you obtain a similar parameterization and fit to the hold-out set across all k folds, then you have some evidence that the model is not overfitting. $\endgroup$
    – ericgtaylor
    Commented Aug 3, 2017 at 13:52
  • $\begingroup$ If you are really concerned about estimating your parameters (if the model is intended to represent a theory, e.g., for academic publication), then I would consider learning about Bayesian estimation. A Bayesian model returns an posterior distribution of your model parameters, and the variance of that distribution corresponds to your uncertainty of the parameter estimates. If you find that the posteriors are extremely wide, then you might have an oversaturated model. $\endgroup$
    – ericgtaylor
    Commented Aug 3, 2017 at 13:56
0
$\begingroup$

I am not sure if this would help, but I believe that Section 4.4 of Harrell's (2015) Regression modeling strategies has an answer to your question. I believe that the book suggests that there should be at least 15 times more observations than coefficients. Please check out the section yourself. I am not certain whether my understanding of the text is correct because I am not an expert.

I also just come across a relevant forum post that recommends this conference paper which may shed light on your problem.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm having trouble finding a copy of that section to read, but from some of Harrell's online course notes and the conference slides you linked to, there may be some insight. The conference slides indicate that 'observations' are nested within 'independent sampling units', however I'm still not sure whether Harrel's suggestion is based on the total number of observations, or the number of observations nested within each independent sampling unit. I'll keep trying to track down a copy of his book to see if the answer is within! Thanks for your help. $\endgroup$
    – dmacnearney
    Commented Aug 1, 2017 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.