46 votes

Assumptions of multiple regression: how is normality assumption different from constant variance assumption?

1. Normal distribution of residuals: The normality condition comes into play when you're trying to get confidence intervals and/or p-values. $\varepsilon\vert X\sim N (0,\sigma^2 I_n)$ is not a ...
Antoni Parellada's user avatar
43 votes
Accepted

Why are survival times assumed to be exponentially distributed?

Exponential distributions are often used to model survival times because they are the simplest distributions that can be used to characterize survival / reliability data. This is because they are ...
klumbard's user avatar
  • 1,391
38 votes
Accepted

How incorrect is a regression model when assumptions are not met?

What happens if the residuals are not homoscedastic? If the residuals show an increasing or decreasing pattern in Residuals vs. Fitted plot. If the error term is not homoscedastic (we use the ...
JohnK's user avatar
  • 20.2k
27 votes

What are the dangers of violating the homoscedasticity assumption for linear regression?

Homoscedasticity is one of the Gauss Markov assumptions that are required for OLS to be the best linear unbiased estimator (BLUE). The Gauss-Markov Theorem is telling us that the least squares ...
Simon O'Rourke's user avatar
26 votes

Family of GLM represents the distribution of the response variable or residuals?

The family argument for glm models determines the distribution family for the conditional distribution of the response, not of the residuals (except for the quasi-models). Look at this way: For the ...
kjetil b halvorsen's user avatar
25 votes

What are the assumptions of factor analysis?

Input data assumptions of linear FA (I'm not speaking here about internal assumptions/properties of the FA model or about checking the fitting quality of results). Scale (interval or ratio) input ...
ttnphns's user avatar
  • 57.2k
23 votes

Assumptions to derive OLS estimator

You can always compute the OLS estimator, apart from the case when you have perfect multicollinearity. In this case, you do have perfect multilinear dependence in your X matrix. Consequently, the full ...
Simon Degonda's user avatar
23 votes

Why are survival times assumed to be exponentially distributed?

To add a bit of mathematical intuition behind how exponents pop up in survival distributions: The probability density of a survival variable is $f(t) = h(t)S(t)$, where $h(t)$ is the current hazard ...
juod's user avatar
  • 2,240
22 votes
Accepted

Does the assumption of Normal errors imply that Y is also Normal?

The standard OLS model is $Y = X \beta + \varepsilon$ with $\varepsilon \sim \mathcal N(\vec 0, \sigma^2 I_n)$ for a fixed $X \in \mathbb R^{n \times p}$. This does indeed mean that $Y|\{X, \beta, \...
jld's user avatar
  • 20.1k
22 votes
Accepted

Are 50% confidence intervals more robustly estimated than 95% confidence intervals?

This answer analyzes the meaning of the quotation and offers the results of a simulation study to illustrate it and help understand what it might be trying to say. The study can easily be extended by ...
whuber's user avatar
  • 321k
22 votes
Accepted

Multicollinearity and predictive performance

Let's assume that you have trained a model on a training dataset, and want to predict some values in a test/holdout dataset. Multicollinearity in your training dataset should only reduce predictive ...
mkt's user avatar
  • 18.1k
21 votes

Family of GLM represents the distribution of the response variable or residuals?

Further to Kjetil's excellent answer, I wanted to add some specific examples to help clarify the meaning of a conditional distribution, which can be a bit of an elusive concept. Let's say you took a ...
Isabella Ghement's user avatar
21 votes
Accepted

Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?

The operational meaning of the IID condition is given by the celebrated "representation theorem" of Bruno de Finetti (which, in my humble opinion, is one of the greatest innovations of ...
Ben's user avatar
  • 123k
20 votes

Why does including $x\ln(x)$ interaction term in logistic regression model helps to assess linearity assumption?

Box and Tidwell (1962) [1] presented a somewhat general approach for estimating transformations of the individual predictors (IVs), and work through the specific case of estimating power ...
Glen_b's user avatar
  • 281k
20 votes
Accepted

What is the need of assumptions in linear regression?

You are correct - you do not need to satisfy these assumptions to fit a least squares line to the points. You need these assumptions to interpret the results. For example, assuming there was no ...
rinspy's user avatar
  • 3,360
19 votes

What is a complete list of the usual assumptions for linear regression?

The following diagrams show which assumptions are required to get which implications in the finite and asymptotic scenarios. Linear Regression Assumptions: Key Points Generally the assumptions can ...
DVL's user avatar
  • 291
19 votes
Accepted

Regression: why test normality of overall residuals, instead of residuals conditional on $\hat{y}$?

Couldn't we have normal residuals at each predicted value of y, while having overall residuals that were quite non-normal? No -- at least, not under the standard assumption that the variance of the ...
Jake Westfall's user avatar
19 votes
Accepted

Why do I see a pattern in the residuals in this well specified model?

How close the residuals at specific $x$ values are to zero depends on the sample size. Now, the sample size in real examples will be whatever it is, so there's not much use saying it should be bigger, ...
Thomas Lumley's user avatar
19 votes

Which OLS assumptions are colliders violating?

I will assume models without intercepts to have shorter notation. Say the structural causal model is \begin{aligned} Y&=\beta_1X+u, \\ Z&=\gamma_1X+\gamma_2Y+v, \\ X&=w \end{aligned} with $...
Richard Hardy's user avatar
18 votes
Accepted

Why is high positive kurtosis problematic for hypothesis tests?

heard [...] that a high positive kurtosis of residuals can be problematic for accurate hypothesis tests and confidence intervals (and therefore problems with statistical inference). Is this true and, ...
Glen_b's user avatar
  • 281k
18 votes
Accepted

Do we really need to include "all relevant predictors?"

You are right - we are seldom realistic in saying "all relevant predictors". In practice we can be satisfied with including predictors that explain the major sources of variation in $Y$. In the ...
Frank Harrell's user avatar
18 votes
Accepted

Is there i.i.d. assumption on logistic regression?

From your previous question you learned that GLM is described in terms of probability distribution, linear predictor $\eta$ and link function $g$ and is described as $$ \begin{align} \eta &= X\...
Tim's user avatar
  • 138k
18 votes

What is the need of assumptions in linear regression?

Try the image of Anscombe's quartet from Wikipedia to get an idea of some of the potential issues with interpreting linear regression when some of those assumptions are clearly false: most of the ...
Henry's user avatar
  • 38.8k
18 votes
Accepted

For linear regression, why do people usually standardize the X variables and log transform Y variables to make them normally distributed?

All 3 of your points are incorrect. The log transformation does not necessarily make the dependent variable normally distributed. But more importantly, it doesn't matter if the dependent variable is ...
mkt's user avatar
  • 18.1k
17 votes
Accepted

What are the consequences of having non-constant variance in the error terms in linear regression?

The consequences of heteroscedasticity are: The ordinary least squares (OLS) estimator $\hat{\mathbf{b}} = \left(X'X \right)X'\mathbf{y}$ is still consistent but it is no longer efficient. The ...
Matthew Gunn's user avatar
  • 22.3k
17 votes

Does regression work on data that isn't normally distributed?

A regression analysis assumes that the data is normally distributed conditioned on the variables in the regression model. That is, if this is the regression model: $$y=X\beta+\varepsilon$$ where $X$ ...
Ruben van Bergen's user avatar
15 votes

How should I check the assumption of linearity to the logit for the continuous independent variables in logistic regression analysis?

Logistic regression does NOT assume a linear relationship between the dependent and independent variables. It does assume a linear relationship between the log odds of the dependent variable and the ...
user114667's user avatar
15 votes

Are 50% confidence intervals more robustly estimated than 95% confidence intervals?

This is an interesting idea, and I can see how it is intuitively compelling, but I think it is too vague to be true or false. Here are a couple of questions I would want the commenter to clear up: ...
gung - Reinstate Monica's user avatar
15 votes

Is the inference from a parametric test valid when the population distribution is not normal?

I need to correct a number of mistaken or partly misplaced ideas in the question first (as well as some that aren't in the question but are commonly seen and may be indirectly influencing the way you ...
Glen_b's user avatar
  • 281k
15 votes
Accepted

What are the k-means algorithm assumptions?

This is a complicated question, as I believe that the role of model assumptions in statistics is generally widely misunderstood, and the situation for k-means is even less clear than for many other ...
Christian Hennig's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible