3
$\begingroup$

I'm modeling a quaternary response using multinomial logit. N is about 3000, however, there are only 500 occurrences of the rarest event.

My usual go-to sources (Agresti 2013: 207; Hosmer et al 2013: 407-408) suggest as a rough guideline that the number of parameters in the model should not exceed 0.1 * the frequency of the rarest event.

All my predictors are categorical, ranging from binary to quinary. My current model has an 1+15 parameters which, multiplied by the 4-1 regression equations, yields a total of 48 parameters. If I were to follow the guideline blindly, I would not be able to add any more variables without the parameter count exceeding the recommended threshold.

But the aforementioned sources are discussing the events-per-parameter question in the context of binary logistic regression. Since a multinomial model for C categories consists of C-1 binary logistic regressions -- each pairing one outcome category vs. the chosen baseline category -- my intuition suggests that the events-per-parameter requirement for the multinomial model should be the same as that of the individual binary logistic models of which it consists. Specifically, it should be the same as the events-per-parameter requirement for that one of the C-1 binary logits in which the rarest event is involved. This page agrees, but its sources do not seem to address the question explicitly.

When judging how many parameters can be estimated by a multinomial model, is the relevant number the events per total parameters over all regression equations or, as I suspect, events per parameter per regression?

Secondly, regarding unbalanced categorical predictors, Hosmer et al (2013: 408) state:

We think that the ten events per parameter rule may be a good conservative working strategy for models with continuous covariates and discrete covariates with a balanced distribution over its categories. However, we are less certain about its applicability in settings where the distribution of discrete covariates is weighted heavily to one value, as often is the case in practice. Here one may require that the minimum observed frequency be, say, 10 in the contingency table of outcome by covariate.

Unless I'm mistaken, in my case this means having to cross-tabulate the response variable separately with each of the 15 predictors. When I do so, I find that there are a total of six predictor categories with <10 observations for one or other response category. Three of these sparse cells are zeroes. What are the implications for the validity of my model?

All input, including references to helpful sources, are much appreciated.

References

Agresti, A. (2013) Categorical Data Analysis. 3rd ed. Hoboken: Wiley.

Hosmer, D. W., Lemeshow, S. & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). Hoboken, N.J.: Wiley.

$\endgroup$
1
$\begingroup$

This is an open area of research. It helps to establish that the multinomial models are just sequences of logistic regression models that have a pooled calculation of deviance. With logistic regression, there are reasons to evaluate the sample size: 1) to achieve adequate power, and 2) to avert small sample bias.

With all models, if the sample size is small, the power will be low. However, unlike linear regression, point estimates in logistic regression models are biased in underpowered analyses. Small sample bias describes the tendency for log odds ratios in logistic regression to bias by a factor of 2. This is discussed in Breslow and Day's IARC publication Statistical Methods in Cancer Research as a justification for conditional analyses.

Small sample bias is a serious issue. It should upturn the laissez-fare research approach of "run the underpowered analysis and let the confidence intervals summarize the uncertainty, we can just pool the analyses in a meta-analysis later on". Alas, we rarely see criticisms of obviously biased ORs in papers and publications. I don't know whether current meta-analysis methods are set-up to handle this type of issue.

One of the earliest and most cited articles addressing small sample bias is Peduzzi, et al 1996 who talks about the number of events-per-variable (EPV) as a metric to evaluate logistic regression. It turns out the Hosmer Lemeshow criterion of inspecting cross-tabulated outcome frequencies is too stringent. Peduzzi gives 15 as an EPV that he believed would mitigate small sample bias. This means that with 30 events (and presumably a larger number of non-events), you could confidently model two predictors even if they are highly collinear. It turns out the EPV is neither sufficient nor necessary. EPV has been refuted several times in the literature. One nice article from Ewout Steyerberg and Peter Austen states this with some authority and they provide nice alternative approaches. I think these are relevant to your question.

Polytomous logistic regression is just sequences of logistic models. If any one of the categorical comparisons suffers the small-sample issues, it will lead to biased estimates for that factor level. Conveniently, similar approaches outlined in the Steyerberg and Austen article can be applied to assess bias and power.

Their recommendation: bootstrap the data, inspect the sampling distribution of the ORs. If they show heavy skewness and heaping due to discretization, the analysis is underpowered and the estimates are significantly biased. In general you cannot simply look at a crosstabular frequency and immediately assess the precision or bias of these estimates. This is especially problematic with 2 or more predictors. The advantage of a linear model is that one is able to borrow information across groups, so that if two or more predictors are strongly collinear, accurate predictions and inference can still be obtained.

$\endgroup$
  • $\begingroup$ I did as you suggest. In a boostrap analysis with 1000 iterations, the only coefficients with serious instability are ones pertaining to predictor categories with low or zero event counts for one or other response category. This would seem to support H&L's view that in the case of categorical covariates, having enough events in all predictor categories is more relevant than events per parameter. $\endgroup$ – blokeman Mar 9 '18 at 19:06
  • $\begingroup$ @blokeman Thanks for reporting the results of your investigation. I would agree that, if my hand were forced to settle on merely one approach to the problem, I would prefer H&L over Peduzzi. But neither is sufficient to explain the problem. Out of curiosity, did you calculate the retrospective power of your analysis (akin to using a parametric bootstrap)? $\endgroup$ – AdamO Mar 9 '18 at 19:08
  • $\begingroup$ Nope, never heard about such a thing in my life :) Anyway, thanks a lot for your clarifications and input. $\endgroup$ – blokeman Mar 9 '18 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.