0
$\begingroup$

This is a general question, which has come up in two recent analyses. I have two datasets wherein a data point was very uncommon but ended up being significant on the analysis.

In one it was female sex that was tied to the outcome (blood clots) in a cohort that included about 500 males and 4 females. In the other it was use of a medication that was tied to the outcome (low bone density) but only 6% of patients were on this medication.

Is there an established rationale for excluding these variables when regression models include them as significant?

$\endgroup$
1
  • 1
    $\begingroup$ Short answer: no. Any general rule would clearly be incorrect in the wrong circumstances. $\endgroup$
    – whuber
    Apr 24, 2023 at 13:11

1 Answer 1

1
$\begingroup$

Frank Harrell's Regression Modeling Strategies provides a lot of guidance on how to design regression models. As @whuber nicely puts it in a comment, "Any general rule would clearly be incorrect in the wrong circumstances." This essentially comes down to the tradeoffs that you are willing to make. A few principles to apply come to mind.

First, omitting any outcome-associated predictor is typically not a good idea, as it leads to a risk of omitted-variable bias. In ordinary linear regression, that will occur if the omitted variable is correlated with the included predictors. In survival analysis and binary regression, omitting any outcome-associated predictor can lead to downward bias in the magnitudes of coefficient estimates for included predictors.

Second, although the first point would suggest including all potentially outcome-associated predictors, as you increase the number of predictors you run a risk of overfitting: getting a model that very well predicts your particular data set but doesn't generalize well to other data samples. I might even worry a bit that, with some very rare predictors "being significant on the analysis," you might already be overfitting your data. Make sure that you have done careful validation of the model before you put too much faith in those results.

Third, the formula for the estimated variance of the estimated linear regression coefficient for predictor $j$, $\hat \beta_j$, provides some clues. From Wikipedia:

$$ \widehat{\operatorname{var}}(\hat{\beta}_j) = \frac{s^2}{(n-1)\widehat{\operatorname{var}}(X_j)}\cdot \frac{1}{1-R_j^2},$$

where $\widehat{\operatorname{var}}(X_j)$, is the estimated variance in predictor $j$, $R_j^2$ is the multiple $R^2$ of the regression of predictor $j$ on the other predictors, $n$ is the number of observations, and $s^2$ is the mean squared residual error.

If you have a rare binary predictor, $\widehat{\operatorname{var}}(X_j)$ is necessarily very small. For a fraction $f_j$ having the rare value of predictor $j$, it's $f_j(1-f_j)$. In your example of 4 females and 500 males, that's only 0.008. That's much less than the maximum variance at a 50/50 distribution, 0.25. The formula means that if you have a rare predictor you will only find a "significant" association with outcome if the magnitude of its association is very large and its correlation with other predictors is small.

That helps define the tradeoffs. Do you want to miss a potentially large association of predictor $j$ with outcome by omitting it from the model? Or do you want to risk overfitting by including more predictors in the model than the size of your data set can reasonably allow?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.