2
$\begingroup$

Let’s say I want to build a linear regression model to conduct some sort of statistical inference. I plan on using the least squares method to fit the model. My understanding is that you need to specify a functional form of the model without looking at your Y values. If you were to look at the Y values your model would be biased. So let’s say I make my best guess as to what the functional form of the true data generating process is. Then I fit the model. Then I look at some diagnostic plots to test the least squares method assumptions. I see that the assumptions are not met. Say the residuals exhibit heteroskedasticity. At this point it is clear that I don’t have a great model. I could use this information to devise a new model. But if I were to fit that, the model would be biased.

Is it possible to build an unbiased model at this point? If so, what are my options? Or should I just accept that any new models I devise will be biased?

$\endgroup$
1
$\begingroup$

Your caution is warranted. Once you have used the outcome data to adjust how you formulate your model, the assumptions underlying p-values and confidence intervals have been violated.*

The principles of how to proceed are explained and illustrated in Frank Harrell's course notes and book. Use a strategy that avoids those problems from the start.

The best way to proceed is (1) to use a type of model that respects the expected error distribution in the outcome once the predictors have been taken into account, and (2) to model the predictors as flexibly as is consistent with the size of your data sample.

Point 1. A standard ordinary least-squares (OLS) regression model is fine if, based on your understanding of the subject matter, the errors around the predictions will be homoskedastic. But if you already know that measurement errors in your strictly-positive outcome value tend to be proportional to the true value (often the case in practice), then modeling the log of the outcome or using a generalized linear model with a log link would be better.

If you are modeling counts, then a Poisson generalized linear model will work better than OLS. Yes/no outcomes can be better handled with logistic regression. For continuous outcomes where you can't make assumptions about error distributions, ordinal regression provides a robust approach that doesn't depend on such assumptions; see Harrell's notes and book.

Choosing the type of model appropriately first will tend to avoid surprises that come from starting blindly with OLS without regard to the nature of the outcome and its measurement.

Point 2. The section of on Multivariable Modeling Strategies in Harrell's notes nicely addresses how to handle predictors. In outline:

(a) Estimate how many degrees of freedom (df) you have to spend on your data without overfitting the size of your data set. For example, with a continuous outcome you might be able to spend 1 df for every 10 or so cases.

(b) Choose which predictors to spend those df on. For categorical predictors, you spend 1 df per level beyond the reference. A continuous variable included as a linear predictor only spends 1 df. Linearity isn't a safe assumption, however, so spend more df for flexible fitting, for example with restricted cubic splines. That way the data themselves will tell you the shape of the association and you don't have to guess the form (and later correct it) on your own.

(c) Run the model. You haven't cheated by looking at the outcomes first, so you have avoided the problems you fear.

Harrell's course notes and book contain many examples of how to apply these principles in practice.


*The types of erroneous modeling that you describe in the question don't necessarily lead to bias in the technical sense of the expected value of model predictions being systematically different from the value in the population. They do, however, wreak havoc with inference based on p-values and confidence intervals. Erroneous modeling is likely to greatly overstate the "statistical significance" of the findings.

$\endgroup$
0
$\begingroup$

The basic linear regression model is $y=X\beta+\epsilon$, with $\epsilon\sim N(0, \sigma^2I)$. We also assume that $\forall i\neq j, Cov(\epsilon_i,\epsilon_j)=0$.

There's a few methods for assessing model assumptions, some of them are explained here. I'll try to give you some key ideas, per violation, but first let us define some terms:

  • Projection matrix of $X$: $P_X=X(X^TX)^{-1}X^T$
  • Error: $e_i=y_i-\hat{y}_i$
  • Standardized error: $e^*_i=e_i/\sigma\sqrt{I_n-P_X}$

So, possible violations:

  1. $E[e]\neq 0$
  2. Non-linear model
  3. $Var(\epsilon)\neq \sigma^2I_n$
  4. Non-normal error

A common method for diagnosing model violations is using plots. A very handy plot is the SRVF (standardized residuals vs. fit): We can see whether violations 1,2,3 occur: Standardized Residuals vs. Fit, origin unknown

Another nice plot is the Q-Q plot, from which we can deduce regarding violation 4.

Now, what do we do if a violation was found?

  1. $E[e]\neq 0$ - adapt your intercept.
  2. Non-linear model - for each covariate $x_j$, plotting $y_i$ vs. $x_j$. If the relation seems like (for example) $y_i \propto x_2^2$, simply add $x_2^2$ as another covariate. Should you end up adding this covariate, you might want to consider using a variable selection method.
  3. $Var(\epsilon)\neq \sigma^2I_n$ - use a variance-stabilizing transformation.
  4. Non-normal error - same as 3.

You might also like to test for outliers and/or influential points in your data and handle them properly.

$\endgroup$
1
  • $\begingroup$ I don’t think this addresses the underlying issue. If you realize the assumptions are violated after specifying and fitting the model, then re-fitting a new model will result in confidence intervals and significance tests that do not perform as expected. See: Berk, R. A., Brown, L., & Zhao, L. (2010). Statistical inference after model selection. Journal of Quantitative Criminology, 26, 217–236. And also: Leeb, H., & Pötscher, B. M. (2006). Can one estimate the conditional distribution of post-model-selection estimators? The Annals of Statistics, 34(5), 2554–2591. $\endgroup$
    – Brian_E
    Jul 25 '21 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.