1
$\begingroup$

In order to run a simple linear model (e.g. using lm() function in R) I am under the impression that the following assumptions must be met:

  1. Normality of residuals
  2. Homoscedasticity
  3. No collinearity (independent variables independent of one another)
  4. No evidence of serial-autocorrelation
  5. No evidence of unduly influential observations
  6. Linear relationship between X and Y
  7. All observations of Y are independent of one another

Would I also be correct in assuming that assumptions 3, 6 and 5 are irrelevant when all independent variables in the model are categorical? In particular, with regards to assumption 5, I thought that these observations can be identified by calculating Cooks distance for each observation and seeing whether or not it exists above a threshold value. Cooks distance requires leverage in order to be calculated however I am confused how this can be calculated when the independent variables are not continuous?

I apologize if this is a silly question as my knowledge of statistics is mostly self-taught.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Assumption #3 is not an assumption of linear modeling. Whether the features are independent or not, the Gauss-Markov theorem is in play, so we get our minimum-variance linear unbiased estimator. The OLS estimator coincides with maximum likelihood estimation if $iid$ conditional Gaussian distributions are assumed, so our p-values and confidence intervals do what they claim to do. Yes, there can be inflation of standard errors due to feature dependence, but this strikes me as a feature, not a bug, of regression modeling: if the features are related, untangling them should be hard.

Regarding assumption #5, even if a feature is continuous, it can have extreme observations that still satisfy the linear trend, so this does not make sense as an assumption, regardless of the features.

Assumption #6 still matters, but we should expect to get silly estimates if we fit a linear model to a nonlinear trend.

Depending on what you are doing, the other proposed assumptions can have varying levels of importance. If you only care about predictions, for instance, you might not care about Gaussian errors.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In regards to assumption 3, would we not need to check for collinearity between independent variables before running a linear model? If so how does this work for categorical variables. $\endgroup$
    – Insect_biologist
    Commented Jan 17, 2023 at 19:06
  • $\begingroup$ @Insect_biologist Many sources will say that a lack of feature correlation is important, and that mostly seems to come from misunderstanding what lack of correlation is required by the Gauss-Markov theorem and a (mistaken, in my opinion) belief that there is so much redundant information in correlated features that one can be dropped in the name of parsimony. Yes, lowering the parameter count has advantages, but dropping parameters has disadvantages, too, chiefly that it can result in estimation bias. $\endgroup$
    – Dave
    Commented Jan 17, 2023 at 19:10
  • $\begingroup$ Regarding the last paragraph: I think that for interval or density predictions, we need to know the error distribution, as we do for point predictions if they are to be tailored to the evaluation loss function so as to minimize the expected loss (and why shouldn't they be?). $\endgroup$
    – Richard Hardy
    Commented Jan 17, 2023 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.