5
$\begingroup$

I consulted various books and get confused about what the differences in Assumptions for Regression Models, Ordinary Least Square (OLS), and Multiple Regression Models are?

As I read more about it I just get more confused. Is there a rationally understandable way to explain this that would not be confusing?

Do these assumptions vary with respect to objective (maybe prediction or descriptive)?

What would be the reasonable list of assumptions for

  1. Linear Regression Models
  2. Simple Linear Regression Models
  3. Multiple Linear Regression Models
  4. Ordinary Least Square (OLS) Method?

I have also read the related questions on What are the Complete List.... and Assumption Needed for Multiple linear Regression. But still it is not clear to me.

$\endgroup$
6
$\begingroup$

It would be difficult to be clearer than what has been said for the other posts. Nevertheless I will try to say something to the point that addresses the different assumptions that are needed for OLS and various other estimation techniques to be appropriate to use.

OLS estimation: This is applied in both simple linear and mutliple regression where the common assumptions are (1) the model is linear in the coefficients of the predictor with an additive random error term (2) the random error terms are (a) normally distributed with 0 mean and (b) a variance that doesn't change as the values of the predictor covariates (i.e. IVs) change, Note also that in this framework which applies in both simple and multiple regression the covariates are assumed to be known without any uncertainty in their given values. OLS can be used when either A) only (1) holds with 2(b) or B) both (1) and (2) hold.

If B) can be assumed OLS has some nice properties that make it attractive to use. (I) MINIMUM VARIANCE AMONG UNBIASED ESTIMATORS (II) MAXIMUM LIKELIHOOD (III) CONSISTENT AND ASYMPTOTICALLY NORMALITY AND EFFICIENCY UNDER CERTAIN REGULARITY CONDITIONS

Under B) OLS can be used for both estimation and predictions and both confidence and prediction intervals can be generated for the fitted values and predictions.

IF only A) holds we still have property (I) but not (II) or (III). If your objective is to fit the model and you don't need confidence or prediction interval for the repsonse given the covariate and you don't need confidence intervals for the regression parameters then OLS can be used under A). But you cannot test for significance of the coefficients in the model using the t tests that are often used nor can you apply the F test for overall model fit or the one for equality of variances. But the Gauss-Markov theorem tells you that property I still holds. However in case A) since (II) and (III) no longer hold other more robust estimation procedures may be better than least squares even though they are not unbiased. This is particularly true when the error distribution is heavytailed and you see outliers in the data. The least squares estimates are very sensitive to outliers.

What else can go wrong with using OLS?

Error variances not homogeneous means a weighted least squares method may be preferable to OLS.

High degree of collinearity among predictors means that either some predictors should be removed or another estimation procedure such as ridge regression should be used. The OLS estimated coefficients can be highly unstable when there is a high degree of multicollinearity.

If the covariates are observed with error (e.g. measurement error) then the model assumption that the covariates are given without error is violated. This is bad for OLS because the criteria minimizes the residuals in the direction of the response variable assuming no error to worry about in the direction of the covariates. This is called the error in variables problem and a solution that takes account of these errors in the covariate directions will do better. Error in variables (aka Deming) regression minimizes the sum of squared deviations in a direction that takes account of the ratios of these variances.

This is a little complicated because many assumptions are involved in these models and objectives play a role in deciding which assumptions are crucial for a given analysis. But if you focus on the properties one at a time to see the consequences of the violation of an assumption it might be less confusing.

$\endgroup$
  • $\begingroup$ The remaining (i.e., error in variables, multicollinearity, heteroscedicity, auto-correlations) are not included in the list of OLS assumptions? $\endgroup$ – Bioinformatics Sep 16 '12 at 18:51
  • 1
    $\begingroup$ I don't know where you get your list but OLS does assume no error in variables, otherwise why would you minimize residuals in the y direction? Homoscedasticity is also a key assumption. The error term is assumed to have a variance that does not depend on the values of the covariates. Variables can be correlated but a high degree of correlation between covariates does cause problems with the coefficients. The error terms for the observations are assumed to be independent of each other. So no autocorrelation is assumed! $\endgroup$ – Michael Chernick Sep 16 '12 at 19:10
  • $\begingroup$ Normality is also considered an assumption. But it only affects the confidence intervals for predictions and coefficients. $\endgroup$ – Michael Chernick Sep 16 '12 at 19:10
5
$\begingroup$

Let me clarify your question:

First, linear regression models comprise all linear models in general. Generally, linear regression models are all about describing the relationship of one variable (dependent) with other variables (independent).

Second, simple and multiple regression models simply refers to the amount of independent variables that one uses in a model. We have a simple regression model in case one uses only one independent variable. In case one uses more than one independent variable to describe a dependent variable than we are calling it multiple regression.

Finally, one can estimate linear regression models in several ways. The most common technique is ordinary least squares (OLS). The OLS method minimizes the sum of squared residuals to estimate the model. It is conceptually simple and computationally straightforward. Other techniques include ML estimations or Bayesian regressions.

That means, we can start talking about the necessary assumptions only once we know what estimation technique we are using to estimate a linear regression model. The only technique you mention in your question is ordinary least squares. You can find a basic understanding of OLS on the following website:

https://economictheoryblog.com/ordinary-least-squares-ols

This site also provides a nice and intuitive description of the assumptions of the OLS estimator:

https://economictheoryblog.com/2015/04/01/ols_assumptions

$\endgroup$
0
$\begingroup$

There's no difference in assumptions for statistical models 1-4. Each one of those models is a form of OLS regression.

The assumptions are the same. The assumptions generally relate to the central limit theorem. If your variables don't have a standard normal distribution then you most likely have a problem.

Common problems: Heteroskedasticity, Multicollinearity, Autocorrelation (time series)

Did this help answer your question?

$\endgroup$
  • 1
    $\begingroup$ Welcome to the site, @Tyler. I appreciate your efforts to help people here, but I worry about what this answer appears to be suggesting. It seems similar to your answer re big data & it might help to clarify this also in light of the comments there. Also, it sounds like you are saying that variables need to be normally distributed. In fact, no variable needs to be normally distributed in regression. You may want to read this: what-if-residuals-are-normally-distributed-but-y-is-not. $\endgroup$ – gung Sep 16 '12 at 18:33
  • $\begingroup$ @gung. Thanks for your advice, I will take it seriously. $\endgroup$ – Tyler Sep 16 '12 at 18:48
  • $\begingroup$ Are these (Heteroskedasticity, Multicollinearity, Autocorrelation) assumptions or problems of OLS? $\endgroup$ – Bioinformatics Sep 16 '12 at 18:53
  • $\begingroup$ Problems due to violations of assumptions except for collinearity. $\endgroup$ – Michael Chernick Sep 16 '12 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.