4
$\begingroup$

I'm fitting a logistic regression model (with multiple predictors) to data where the outcome is a success or failure. My data points are in the range of 100,000. Most of my variables are categorical, but the category sizes are unequal and very skewed - the majority category accounts for > 90% of the data points, and each minority category for about only 1-2% of the data points.

Once I ran the logistic regression model, I obtained the coefficients for each categorical variable, as well as the associated confidence intervals. I learnt how to interpret the coefficients in a logistic regression here, and how to incorporate the confidence intervals in probability calculations here.

The categorical variables corresponding to categories with a smaller number of observations have larger coefficients, which implies a higher probability of success (all else held equal) for that category. The uncertainty resulting from the smaller sample size also seems to be captured by the wider confidence intervals of the coefficients of said variables.

The issue I am facing is this: is it meaningful to compare the coefficients and confidence intervals this way, when most categories have fewer observations ( ~1%) compared to the dominant case? When the coefficients are larger in the minority case, and the confidence intervals (for coefficients, and hence probabilities) are not overlapping, it seems simple to conclude that the minority categories have a higher success rate. This result could be the result of random chance, given the smaller number of observations of these categories.

TL;DR: When dealing with categories which are very skewed in the number of observations, is the variation in confidence intervals a good measure, or is there a better way to account for the size differences for each category while comparing their regression coefficients?

$\endgroup$

1 Answer 1

9
$\begingroup$

It's a good question, and you already have most of an answer: in a set-up like yours it is essential to look at the confidence intervals, not just the coefficient estimates. The machinery is intended to report uncertainty as well as a model fit. Questions about the effect of chance variations are already answered by each confidence interval.

In general,

  1. Many (perhaps most) researchers would want to use all the detail in each variable, unless and until results show that you need to do otherwise.

  2. It is the number of observations in each category that is important, not their proportion in the dataset.

  3. There may be a way of lumping unusual categories together that makes scientific or practical sense and would answer the question of whether results change much with a model using a coarsened version of that variable. (Usual answer: No.)

  4. Do the coefficient estimates for rare categories make sense? This is a substantive question needing subject-matter knowledge.

Detail: I would not use the term skewness here unless any categorical variable concerned is ordinal. I would tend to reach for terms like uneven or concentrated distribution.

$\endgroup$
5
  • $\begingroup$ I think the OP may also need to think about a multiple comparison correction. $\endgroup$
    – dipetkov
    Commented May 20 at 19:19
  • $\begingroup$ Thank you for your answer, I was mainly concerned if the size effects were adequately accounted for by the confidence intervals, or if I had to make further corrections. A quick question, in 2., why do you say number and not proportion matters? I thought a category with relatively fewer observations would be more prone to error in estimating whatever we're seeking, compared to the most abundant category. $\endgroup$ Commented May 21 at 4:31
  • $\begingroup$ @dipetkov I will look into multiple comparison corrections, any suggestions on what would work well for a logistic regression setup? $\endgroup$ Commented May 21 at 4:34
  • $\begingroup$ I've researched a little bit further, and I think pairwise chi-square testing (possibly with Bonferroni corrections) will help establish whether the difference in probabilities for each category is significant. Does that make sense? $\endgroup$ Commented May 21 at 5:05
  • $\begingroup$ It's standard sampling theory that the absolute size of a sample is crucial, not what proportion it is of a population, is crucial, unless the proportion is very high. That's what makes e.g. polling say 2000 people out of tens or hundreds of millions attractive in principle (difficulties in getting random samples are different). $\endgroup$
    – Nick Cox
    Commented May 21 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.