Timeline for Why Normality assumption in linear regression
Current License: CC BY-SA 4.0
37 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 17, 2019 at 11:18 | comment | added | kjetil b halvorsen♦ | @Neil G: I'm the lazy one? You could easily have included surprisal in the original post, yes? Also, when I am making such comments, is is more for the site than for myself. This site is supposed to be self-contained. I could have/did guess the meaning (even if it is nonstandard terminology in statistics), as you can see from my answer here, entropy | |
| Mar 4, 2019 at 0:07 | comment | added | Neil G | Who cares about he marginal distribution? We're not estimating the marginal distribution. And no my point of view doesn't disagree with why the normal distribution is often assumed. Instead of thinking of them as addititive normal errors, you can think of the targets as measurements that induce normal likelihoods, e.g., reading a value on a ruler. | |
| Mar 3, 2019 at 23:45 | comment | added | Sextus Empiricus | Your point of view would also fail with ordinary linear regression, the OP's starting point. "why we choose normal as the distribution that error term follows" | |
| Mar 3, 2019 at 23:41 | comment | added | Sextus Empiricus | You can do that if you like, but then you are just using different language (you make it a semantic discussion, about how we should name things, and not a discussion about ideas). In GLM the distributional assumption relates to the conditional distribution. The marginal distribution is almost never a typical (exponential) distribution. See for instance Gung's plot in the first item that I linked to. | |
| Mar 3, 2019 at 23:39 | comment | added | Neil G | I would call the whole thing the distribution. After all, an exponential family is parametrized by a sufficient statistic, a log-normalizer, a parameter function, a carrier measure, and a support. | |
| Mar 3, 2019 at 23:36 | comment | added | Sextus Empiricus | So the link function $f$ does not relate to the (conditional) distributional assumption. | |
| Mar 3, 2019 at 23:34 | comment | added | Neil G | Yes, in this case, it is normal with unit variance parametrized by $f$ of mean. This is still an exponential family. That's your distributional assumption. | |
| Mar 3, 2019 at 23:34 | comment | added | Sextus Empiricus | Note that we do not speak about the marginal distribution of $y$, but about the conditional distribution of $y$. See for related items: stats.stackexchange.com/questions/12262/… and stats.stackexchange.com/questions/342759/… | |
| Mar 3, 2019 at 23:23 | comment | added | Sextus Empiricus | I expand it as far as one would expand it with a linear model. If you claim that I should expand it further then you should do the same with the linear model and state that OLS is not relating to the assumption of a normal distribution. $$ L = (Z - f(X\beta))^2 = Z^2 - 2Yf(X\beta) + f(X\beta)^2 $$ or you can say $$P(Z|X) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{\frac{-(Z-\theta)^2}{2\sigma^2}}$$ with $\theta = f(X\beta)$ | |
| Mar 3, 2019 at 23:19 | comment | added | Neil G | @MartijnWeterings I understand that many people want to see things in the way you are describing them. However, it's really just a bad way to see things. Do what I said with any of your examples: expand $f$, and then rewrite the equation as an exponential family. That's how you'll find out what distributional assumption you're actually making. Your refusal to actually expand it is what is causing you to miss the connection. | |
| Mar 3, 2019 at 23:18 | comment | added | Sextus Empiricus | You do not need to expand $f$. You neither expand for different linear models. That is, whatever the model for the mean $\beta_0$ or $\beta_0 + x_1 \beta_1$ or $\beta_0 + x_1 \beta_1 + x_2 \beta_2$ or $ \beta_0 + x_1 \beta_1 + x_2 \beta_2 + x_3 \beta_3$ etc. we speak of a normal distribution (conditional on $x$). The same is true when you do non-linear modeling (e.g. non linear least squares). Then the (conditional) distribution (conditional on $x_1, x_2, ... , x_n$) is also normal. | |
| Mar 3, 2019 at 23:15 | comment | added | Sextus Empiricus | @NeilG The use of this language is ubiquitous, I am only using wikipedia as an easy reference, I am not relying on it. Note that I mentioned the original article from Nelder and Wedderburn before. | |
| Mar 3, 2019 at 23:08 | comment | added | Neil G | Regardling your last loss function, you need to expand $f$, and then rewrite the equation as an exponential family. That's how you'll find out what distributional assumption you're actually making. | |
| Mar 3, 2019 at 23:07 | comment | added | Neil G | Your comments are a silly misunderstanding and ignorant insistence on a misconception. Don't rely on the wikipedia entry about GLMs as your source. It's not well-written. | |
| Mar 3, 2019 at 23:01 | comment | added | Sextus Empiricus | @NeilG I hope that you see that my comments are not just a silly misunderstanding and an ignorant insistence on a wrong idea. The use of different link functions with the same distributional assumption is not at all strange language in statistics. So whatever you mean should be clear about the distinction with this common language. For an example of this language see the wikipedia entry about GLM which contains something like "There are several popular link functions for binomial functions" | |
| Mar 3, 2019 at 22:52 | comment | added | Sextus Empiricus | $$L = (Z-f(X\hat\beta))^2$$ | |
| Mar 3, 2019 at 22:45 | comment | added | Neil G | @MartijnWeterings Sorry that I can't be any clearer. Your insistence that the link function is separate from the distributional assumption is wrong. Write out the loss function using your link function and what you think the distributional assumption is; then, you will find the actual distributional assumption. | |
| Mar 3, 2019 at 22:40 | comment | added | Sextus Empiricus | Typically certain link functions are used with certain distributional assumptions. But this is not a necessity. So my distributional assumptions are normal in that example, and not Poisson (that was intentional). Some better (more practical and well known) examples are binomial/Bernouilli distributed variables where people work with a probit model or a logit model, thus different link functions but the same (conditional) distributional assumption. | |
| Mar 3, 2019 at 22:36 | comment | added | Neil G | @MartijnWeterings You mean on $X$ and $\beta$, sure. But actually, your distributional assumptions are not normal. I think your first equation is actually Poisson regression, and corresponds to that distributional assumption. Your second one is something like $f(z) = e^{\eta z - \eta^{\frac32}}$. | |
| Mar 3, 2019 at 22:26 | comment | added | Sextus Empiricus | Sorry I did not wrote it down correctly. It is a normal distribution conditional on the parameters X. $$Z|X \sim N(exp(X\beta),\sigma^2)$$ and $$Z|X \sim N(\sqrt{X\beta},\sigma^2)$$ So they have the same conditional distribution assumption. (more typical examples is the difference probit/logit regression) | |
| Mar 3, 2019 at 22:25 | comment | added | Neil G | @MartijnWeterings Your second equation corresponds to a different distributional assumption, but you just haven't written out what it actually is. It is definitely not normal though. | |
| Mar 3, 2019 at 22:22 | history | edited | Neil G | CC BY-SA 4.0 |
Added some dimensions to make things easier to follow.
|
| Mar 3, 2019 at 22:21 | comment | added | Sextus Empiricus | This makes no sense when the link function can be changed without changing the distributional assumption. For instance, $Z \sim N(exp(X\beta),\sigma)$ and $Z \sim N(\sqrt{X\beta},\sigma)$ are both GLM models that assume a normal distribution, but with different link functions. (maybe we relate to the same thing and this is just a semantic discussion?) | |
| Mar 3, 2019 at 22:17 | history | edited | Neil G | CC BY-SA 4.0 |
Added some dimensions to make things easier to follow.
|
| Mar 3, 2019 at 22:14 | comment | added | Neil G | @MartijnWeterings No, you are wrong. The link function comes from the distributional assumption. I've updated the answer to make this link obvious. The gradient log-normalizer is your $g$. You can find a good synopsis of exponential families by Nielsen and Nock that I highly recommend. | |
| Mar 3, 2019 at 22:13 | comment | added | Sextus Empiricus | The linked article contains in section '3.1 Normal distribution' > "More generally, as shown in Nelder (1968), we can consider models in which there is a linearizing transformation $f$ and a normalizing transformation $g$" I do not know what your gradient log-normalizer refers to, and maybe you are speaking about this normalizing transformation? But, that is not the link function. The link function in GLM relates to the linearizing transformation. | |
| Mar 3, 2019 at 22:11 | history | edited | Neil G | CC BY-SA 4.0 |
added 623 characters in body
|
| Mar 3, 2019 at 21:57 | comment | added | Sextus Empiricus | GLM is the combination (i) a variable z dependent on $\theta$ distributed as$$\pi(z;\theta,\phi)=\exp\left[\alpha(\phi)\lbrace z\theta-g(\theta)+h(z)\rbrace+\beta(\phi,z)\right]$$ (ii) a linear model part$$y_j=\sum\beta_ix_{ij}$$ (iii) a linking function that relates the (natural) parameter $\theta$ to the linear model $$\theta=f(y)$$I am unaware that the link function $f$ needs to be monotonic (I imagine it may indeed be problematic, e.g. for a sinus function). But in any case the link funcion is about point (iii) and not about (i). See: jstor.org/stable/2344614 | |
| Mar 3, 2019 at 21:18 | comment | added | Neil G | @MartijnWeterings No, it needs to be a monotonic function, and I think every monotonic function corresponds to some gradient log-normalizer. | |
| Mar 3, 2019 at 21:17 | comment | added | Sextus Empiricus | You can use any link function that you like in GLM. | |
| Mar 3, 2019 at 21:13 | comment | added | Neil G | @MartijnWeterings "The link function does not have to do with generalizing to different distributional assumptions"--no, it actually does. The link function is the gradient log-normalizer of the assumed distribution. I'll fill in the details later this afternoon. | |
| Mar 3, 2019 at 20:57 | comment | added | Sextus Empiricus | "each link function corresponds to a different distributional assumption" this is very vague. The link function does not have to do with generalizing to different distributional assumptions, but with generalizing the (linear) part that describes the mean of the distribution. | |
| Mar 3, 2019 at 18:49 | comment | added | Neil G | @kjetilbhalvorsen it seems lazy not to just google the terms you're unfamiliar with en.wikipedia.org/wiki/Surprisal | |
| Mar 2, 2019 at 23:03 | history | edited | Neil G | CC BY-SA 4.0 |
deleted 106 characters in body
|
| Mar 2, 2019 at 22:56 | history | edited | Neil G | CC BY-SA 4.0 |
added 1224 characters in body
|
| Mar 2, 2019 at 15:27 | comment | added | kjetil b halvorsen♦ | This is very short and too cryptic for our standards, please also explain surprisal. | |
| Mar 2, 2019 at 15:03 | history | answered | Neil G | CC BY-SA 4.0 |
