46
votes
Assumptions of multiple regression: how is normality assumption different from constant variance assumption?
1. Normal distribution of residuals:
The normality condition comes into play when you're trying to get confidence intervals and/or p-values.
$\varepsilon\vert X\sim N (0,\sigma^2 I_n)$ is not a ...
- 26.9k
43
votes
Accepted
Why are survival times assumed to be exponentially distributed?
Exponential distributions are often used to model survival times because they are the simplest distributions that can be used to characterize survival / reliability data. This is because they are ...
- 1,411
31
votes
Family of GLM represents the distribution of the response variable or residuals?
The family argument for glm models determines the distribution family for the conditional distribution of the response, not of the residuals (except for the quasi-models).
Look at this way: For the ...
- 82.8k
25
votes
Accepted
Multicollinearity and predictive performance
Let's assume that you have trained a model on a training dataset, and want to predict some values in a test/holdout dataset. Multicollinearity in your training dataset should only reduce predictive ...
- 20.4k
24
votes
Why are survival times assumed to be exponentially distributed?
To add a bit of mathematical intuition behind how exponents pop up in survival distributions:
The probability density of a survival variable is $f(t) = h(t)S(t)$, where $h(t)$ is the current hazard ...
- 2,270
23
votes
Family of GLM represents the distribution of the response variable or residuals?
Further to Kjetil's excellent answer, I wanted to add some specific examples to help clarify the meaning of a conditional distribution, which can be a bit of an elusive concept.
Let's say you took a ...
- 20.9k
22
votes
Accepted
Are 50% confidence intervals more robustly estimated than 95% confidence intervals?
This answer analyzes the meaning of the quotation and offers the results of a simulation study to illustrate it and help understand what it might be trying to say. The study can easily be extended by ...
- 334k
22
votes
Accepted
Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?
The operational meaning of the IID condition is given by the celebrated "representation theorem" of Bruno de Finetti (which, in my humble opinion, is one of the greatest innovations of ...
- 133k
20
votes
Accepted
What is the need of assumptions in linear regression?
You are correct - you do not need to satisfy these assumptions to fit a least squares line to the points. You need these assumptions to interpret the results. For example, assuming there was no ...
- 3,390
19
votes
What is a complete list of the usual assumptions for linear regression?
The following diagrams show which assumptions are required to get which implications in the finite and asymptotic scenarios.
Linear Regression Assumptions: Key Points
Generally the assumptions can ...
- 291
19
votes
Accepted
Why do I see a pattern in the residuals in this well specified model?
How close the residuals at specific $x$ values are to zero depends on the sample size. Now, the sample size in real examples will be whatever it is, so there's not much use saying it should be bigger, ...
- 46.8k
19
votes
Which OLS assumptions are colliders violating?
I will assume models without intercepts to have shorter notation. Say the structural causal model is
\begin{aligned}
Y&=\beta_1X+u, \\
Z&=\gamma_1X+\gamma_2Y+v, \\
X&=w
\end{aligned}
with $...
- 69.5k
19
votes
Accepted
For linear regression, why do people usually standardize the X variables and log transform Y variables to make them normally distributed?
All 3 of your points are incorrect.
The log transformation does not necessarily make the dependent variable normally distributed. But more importantly, it doesn't matter if the dependent variable is ...
- 20.4k
18
votes
Accepted
What are the consequences of having non-constant variance in the error terms in linear regression?
The consequences of heteroscedasticity are:
The ordinary least squares (OLS) estimator $\hat{\mathbf{b}} = \left(X'X \right)X'\mathbf{y}$ is still consistent but it is no longer efficient.
The ...
- 23k
18
votes
Accepted
Is there i.i.d. assumption on logistic regression?
From your previous question you learned that GLM is described in terms of probability distribution, linear predictor $\eta$ and link function $g$ and is described as
$$
\begin{align}
\eta &= X\...
- 141k
18
votes
What is the need of assumptions in linear regression?
Try the image of Anscombe's quartet from Wikipedia to get an idea of some of the potential issues with interpreting linear regression when some of those assumptions are clearly false: most of the ...
- 42k
17
votes
Does regression work on data that isn't normally distributed?
A regression analysis assumes that the data is normally distributed conditioned on the variables in the regression model. That is, if this is the regression model:
$$y=X\beta+\varepsilon$$
where $X$ ...
- 7,752
15
votes
Are 50% confidence intervals more robustly estimated than 95% confidence intervals?
This is an interesting idea, and I can see how it is intuitively compelling, but I think it is too vague to be true or false. Here are a couple of questions I would want the commenter to clear up:
...
15
votes
Accepted
What are the k-means algorithm assumptions?
This is a complicated question, as I believe that the role of model assumptions in statistics is generally widely misunderstood, and the situation for k-means is even less clear than for many other ...
- 28.3k
14
votes
What does "independent observations" mean?
In probability theory, statistical independence (which is not the same as causal independence) is defined as your property (3), but (1) follows as a consequence$\dagger$. The events $\mathcal{A}$ and ...
- 133k
14
votes
Is the inference from a parametric test valid when the population distribution is not normal?
I need to correct a number of mistaken or partly misplaced ideas in the question first (as well as some that aren't in the question but are commonly seen and may be indirectly influencing the way you ...
- 290k
14
votes
Why linear regression has assumption on residual but generalized linear model has assumptions on response?
The assumptions are not inconsistent. If, for $i = 1, \ldots, n$, you assume
$$
Y_i = \beta_0 + \beta_1 X_{i1} + \ldots + \beta_k X_{ik} + \epsilon_i,
$$
with the errors $\epsilon_i$ being normally ...
- 3,298
14
votes
Accepted
Why linear regression has assumption on residual but generalized linear model has assumptions on response?
Simple linear regression having Gaussian errors is a very nice attribute that does not generalize to generalized linear models.
In generalized linear models, the response follows some given ...
- 21.6k
14
votes
Instrumental variable exclusion restriction
There are two criteria for good instruments:
The instrument $z$ is correlated with the endogenous variable $x$ (relevance).
The instrument $z$ affects dependent variable $y$ only through $x$. In ...
- 268
14
votes
Accepted
What are the assumptions for applying a quantile regression model?
Quantile regression assumes
the normal regression assumptions of linearity and additivity (unless you add more terms to the model)
independence of observations
very large sample size, as quantile ...
- 98.4k
14
votes
Model assumptions - not worth the effort?
Well, there is some truth to it, but it's problematic.
It would lower the publication rate. OK. But getting rid of bad publications is a good thing. Too much garbage is published.
Checking model ...
- 128k
14
votes
Are data in the real world "sampled" in the statistical sense?
Thich Nhat Hanh said that "everything depends on everything else". He is probably right, meaning that in reality nothing is really i.i.d.
Note also that probability can be used with more ...
- 28.3k
13
votes
Why are survival times assumed to be exponentially distributed?
You'll almost certainly want to look at reliability engineering and predictions for thorough analyses of survival times. Within that, there are a few distributions which get used often:
The Weibull (...
- 271
13
votes
Accepted
Kruskal-Wallis test: assumption testing and interpretation of the results
The KW test (also the Mann-Whitney U-test) is essentially always a test for stochastic dominance. What that means is it is testing to see if there exists at least one group such that you would ...
13
votes
Accepted
What are the consequences of rare events in logistic regression?
The standard rule of thumb for linear (OLS) regression is that you need at least $10$ data per variable or you will be 'approaching' saturation. However, for logistic regression, the corresponding ...
Only top scored, non community-wiki answers of a minimum length are eligible