6
$\begingroup$

Suppose we have a linear regression model $Y_{it} = \beta_0 + \beta_1 X_{it} + \epsilon_{it}$, many times in literature it is assumed that $\epsilon_{it} \sim N(0,\sigma^2).$ This assumptions makes sense if we have a large data set due to the central limit theorem. My questions is that in certain situations I feel the error term being normally distributed is the wrong assumption. Suppose $Y_{it}$ is a bounded variable, such as age of a person, or a exam score of a student. Then if $\epsilon_{it} \sim N(0,\sigma^2)$ in this situations where $Y_{it}$ is bounded, is it not possible for the error term to be such that it forces $Y_{it}$ out of its bounds? For example suppose $Y_{it}$ represents a persons age, if the error term is normally distributed, then a random event could occur so it is possible for a person to live say a 1000 years?

Hence, how do we fix this issue with the error term when our dependent variable on the left side on the linear equation is bounded. We could choose another bounded distribution for the error term, such as the uniform distribution over the bounds of $Y_{it}$. However this would not be realistic since it would imply all events in the error term are equally likely to occur. I am interested to here people thoughts about this problem.

Edit: From reading all the great answers and comments below, here is what I have to say. Would it be practical to impose a bounded domain distribution on $\epsilon_{it}?$ For example the triangle density over a particular domain that $Y_{it}$ is in. Would imposing these types of distribution which have a bounded domain and resemble the normal distribution have any disadvantages?

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ you can't gain anything from imposing arbitrary conditions on error terms. you gain something only if your new conditions better reflect the reality. $\endgroup$ – Aksakal Apr 8 '14 at 13:56
  • $\begingroup$ @Aksakal, I see what you mean, and the normal distribution represents reality, that's why we use it? Suppose we were to impose a bell curve shape distribution to $\epsilon_{it}$ that had the appropriate bounded domain. Would this be a good solution to the problem? Suppose $Y_{it}$ was a person's age, and we choose a nice bell curve distribution for $\epsilon_{it}$ that had domain $[0,150]$, this would be better than just using the normal distribution for $\epsilon_{it}?$ $\endgroup$ – user77404 Apr 8 '14 at 14:00
  • $\begingroup$ if normal distribution assumption seems reasonable, then you use it. you can also test the assumption using Jarque-Bera or similar tests. in physical sciences normal assumption often works very well. there are bounded distributions, such as beta distribution. frankly, i don't understand what is the problem to solve here. bounded distributions have their own issues. the point is to come up with the distribution which reflects the reality, in your case it could be lognormal, for instance $\endgroup$ – Aksakal Apr 8 '14 at 14:04
  • $\begingroup$ @Aksakal, your making very good points here. Okay, suppose you were modeling student's performance on test score. Maybe you wanted to see how parental wealth had a impact on student performance, your model was $T_{it} = \beta_0 + \beta_1 W_{it} + \epsilon_{it}$, and the test scores were in range $[0,100]$, what distribution would you use for the error term in this situation. My problem is that using the normal distribution is not realistic, I would suppose a bounded distribution from $[0,100]$ would be better. $\endgroup$ – user77404 Apr 8 '14 at 14:09
  • $\begingroup$ in this case normal assumption may or may not work very well, depending on what is your goal. secondly, you seem to have a panel data set, not the cross-sectional data, which suggest me that MIXED effects regression, instead of simple linear regression. i would start with normal assumption and test it. you always start with simplest stuff and try to make it work before going for fancy things. $\endgroup$ – Aksakal Apr 8 '14 at 14:15
5
$\begingroup$

it seems that you're confused about relation of the sample size to CLT application. the distribution of $\epsilon_{it}$ has nothing to do with the sample size. I'm assuming that subscript $i$ refers to the subject (a person), and a subscript $t$ refers to the tume of othe observation.

in a simple linear regression we don't make a lot of assumptions about $\epsilon$ to estimate $\beta_i$. the errors don't have to be normal, and with increasing sample size they will not tend to become normal.

CLT is applied in two different ways:

  • when a sample size increases then the distribution of an estimate of $\beta_i$ which is often denoted as $\hat{\beta}_i$ will tend to become normal, i.e. $\hat{\beta}_i\sim\mathcal{N}(0,\sigma_\beta)$, where $\sigma_\beta$ is a function of $\sigma$. Again, we do not require $\epsilon_{it}\sim\mathcal{N}(0,\sigma)$, we only need $var[\epsilon_{it}]=\sigma$ for this. This is one of large sample properties of linear regressions.
  • often times when we deal with physical experiments, one could argue that there are many sources of errors, when they all add up, they make $\epsilon_{it}$ - a single observation noise - distributed normally. this is not related to the sample size, this is simply sources of errors influencing a single observation. in this case we often make a reasonable assumption of $\epsilon_{it}\sim\mathcal{N}(0,\sigma)$
| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Depending on the nature of the response variable, I would suggest checking out either the GLM or Tobit models. GLM for when the response is non-normal (eg counts), and Tobit if it could be normal except it is getting censored (eg negative incomes get reported as zero).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ The Tobit model generally also has normally distributed error terms, I'll check out GLM. Suppose our response variable was test score that ranged from 100 to 300, which model would be appropriate here? $\endgroup$ – user77404 Apr 8 '14 at 1:47
2
$\begingroup$

The central limit theorem does not imply that the errors are Normal if you have a large data set. The CLT applies to sums of random variables (under other certain conditions).

As the other poster says, you might look at generalized linear models which allow for non-normal error distributions.

However, note that linear regression does not require normally distributed errors. Regardless of the distribution, the least squares estimator is the Best Linear Unbiased Estimater (BLUE) by the Gauss-Markov theorem. They only need to be uncorrelated and have the same variance.

The normal distribution is only required if you want to claim that the least squares estimate is also the maximum likelihood estimator.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I agree with your comment on the CLT. Then we could pick any bounded distribution the resembles the normal, such as the triangle density. One question I have here is that does the distribution of the error term depend on the data? $\endgroup$ – user77404 Apr 8 '14 at 2:04
  • 1
    $\begingroup$ @user77404, error distribution may or may not depend on data. many nice properties of least squares, such as small sample properties like BLUE, require that error distribution does not depend on data - this s called exogeneity, read about it. unfortunately, in time series this is too strong of a requirement. but something tells me that it's not what you meant in your question. so you may want to clarify it. $\endgroup$ – Aksakal Apr 8 '14 at 13:05
  • $\begingroup$ @Dave31415, to be precise Gauss-Markov theorem requires that error terms are not correlated with the sample (exogeneity) and have mean zero and stable variance - both conditionally, if this is an observational data set, not the experiment. also, normality is necessary not only for equivalence with MLE, but for nice finite sample properties of least squares, otherwise many good things are held only in large samples, i.e. asymptotthically $\endgroup$ – Aksakal Apr 8 '14 at 13:11
  • $\begingroup$ @Aksakal your comment about exogneity makes perfect sense to me now. This clears up my concern about the error distribution depending on the data or not. $\endgroup$ – user77404 Apr 8 '14 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.