12
$\begingroup$

I am learning about linear regression from the textbook Elements of Statistical Learning by Friedman, Tibshirani, and Hastie. In this section they suppose we have a set of training data $(x_1, y_1), \ldots, (x_N, y_N)$ where each $x_i \in \mathbb{R}^p$ for some $p$ and $y_i \in \mathbb{R}$. We wish to find $\hat{\beta}$ that best estimates the regression parameters $\beta$ by using least squares.

I am confused about the following passage in the textbook:

Up to now we have made minimal assumptions about the true distribution of the data. In order to pin down the sampling properties of $\hat{\beta}$, we now assume that the observations $y_i$ are uncorrelated and have a constant variance $\sigma^2$, and that the $x_i$ are fixed (non random).

The $y_i$ are just a set of real numbers, how can it not have constant variance and how can it be correlated? Why are these assumptions necessary to make any claims about the distribution of $\beta$?

$\endgroup$
2

3 Answers 3

10
$\begingroup$

These are assumptions made for certain models to ensure certain properties, like valid test statistics. There's a great overview here. The key word here is assumption. These need not hold up in real data, but are often reasonable approximations.

Independence: Observations are uncorrelated if the value of the outcome $y_i$ does not depend on the value of other observations. For example:

  • $\color{green}{\text{Independent}}$: You have observations of height from 30 randomly selected individuals. We have no reason to assume one person's height depends on another's, unless they are related.
  • $\color{red}{\text{Dependent}}$: You have observations of 15 individuals' blood pressure before and after some treatment. In this case, whatever someone's blood pressure was before will be correlated to what it turned out to be after treatment.

Constant variance: You assume some probability distribution of errors around your model. In case of ordinary linear regression, a normal distribution. If this normal distribution has the same variance (diffuseness) at any given point, then the variance is constant.

![enter image description here

how can it not have constant variance?

For example, by varying with (one of the) explanatory variables, as shown above.


I made a 5 minute YouTube video explaining these assumptions in the context of simple linear regression. At 3:20, it shows what constant vs non-constant variance is.

$\endgroup$
2
  • 2
    $\begingroup$ This answer appears (to me) to be clarifying concepts that aren't misunderstood by the querent. It is a great explanation of those concepts, but nevertheless isn't really answering the question. $\endgroup$ Commented Jun 25 at 18:19
  • 2
    $\begingroup$ The first part explains how the $y_i$ could be correlated, the second how their variance could be non-constant. Why the assumptions are necessary is explained in the linked post in the first paragraph. $\endgroup$ Commented Jun 25 at 18:38
10
$\begingroup$

Random variables VS observations. Strictly speaking, there are random variables (which take values in $\mathbb{R}$) and realizations of these random variables (which are elements of $\mathbb{R}$). Some authors are explicit about this and write e.g. $Y_i$ for the random variables and $y_i$ for the realizations. Others do not make this distinction clear, as often it is clear from context what is meant.

In that sense, the author means ``the random variables $Y_i$ (which generate the observations $y_i$) are uncorrelated and have constant variance.''

Constant variance. In the context of Regression, ``constant variance'' of $Y_i$ typically refers to `constant with respect to $x_i$'. So, homoscedasticity, as opposed to heteroscedasticity.

$\endgroup$
7
$\begingroup$

$y_i$'s are not just real numbers. They are random variable. Specifically, the simplest linear model assumes $$y_i = x_i^T\beta+\varepsilon_i,\quad \varepsilon_i\overset{iid}{\sim}\mathcal{N}(0,\sigma^2),\quad i=1,\dots,n.$$ If you want they have nonconstant variance, you can relax this assumption. For example, you can assume $$y_i = x_i^T\beta+\varepsilon_i,\quad \varepsilon_i\overset{iid}{\sim}\mathcal{N}(0,\color{red}{\sigma_i^2}),\quad i=1,\dots,n$$ and $\sigma_i$'s can be different from each other.

Moreover, if you want they are correlated, you can throw away indpendent assumption for $\varepsilon_i$'s and put an covariance structure on them. For example, let $Y = (y_1,\dots,y_n)^T$, $X = (x_1,\dots,x_n)^T$, $x_i=(x_{i1},\dots,x_{ip})^T$, $\varepsilon = (\varepsilon_1,\dots,\varepsilon_n)^T$. Then you might write $$Y = X\beta + \varepsilon,\quad \varepsilon\sim \mathcal{N}(\mathbb{0},\color{red}{\Sigma}). $$ This is a more general form of linear models.

Hope this answer is helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.