Help understanding Standard Error

Question

Quoting from An Introduction to Statistical Learning with Applications in R (James, Witten, Hastie, Tibshirani), Chapter 3 on Linear Regression. I used numbered superscripts ^{_1, _2, _3} to mark the areas where I have a corresponding question:

We have established that the average of $\hat{\mu}$ over many data sets will be very close to $\mu$ , but that a single estimate $\hat{\mu}$ may be a substantial underestimate or overestimate of $\mu$. How far off will that single estimate of $\hat{\mu}$ be? In general, we answer this question by computing the standard error of $\hat{\mu}$, written as $SE(\hat{\mu})$. We have the well-known formula:

$Var(\hat{\mu}) = SE(\hat{\mu})^2 = \frac{\sigma^2}{n}$

where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$ (This formula holds provided that the $n$ observations are uncorrelated^_1.) Roughly speaking, the standard error tells us the average amount that this estimate $\hat{\mu}$ differs from the actual value of $\mu$. Equation 3.7 also tells us how this deviation shrinks with $n$—the more observations we have, the smaller the standard error of $\hat{\mu}$^_2. In a similar vein, we can wonder how close $\hat{\beta_0}$ and $\hat{\beta_1}$ are to the true values $\beta_0$ and $\beta_1$. To compute the standard errors associated with $\hat{\beta_0}$ and $\hat{\beta_1}$, we use the following formulas:

$SE(\hat{\beta_0})^2 = \sigma^2\left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i-\bar{x})^2}\right]$

$SE(\hat{\beta_1})^2 = \frac{\sigma^2}{\sum_{i=1}^n (x_i-\bar{x})^2}$

where $\sigma^2 = Var(\epsilon)$. For these formulas to be strictly valid, we need to assume that the errors $\epsilon_i$ for each observation are uncorrelated^_1 with common variance $\sigma^2$. This is clearly not true in Figure 3.1^_3, but the formula still turns out to be a good approximation.

For your reference, here is figure 3.1:

Questions and attempts at answers:

1. I've marked two areas the assumption of "uncorrelated" data appears.

   A. What does it mean for a 1-dimensional dataset to be "correlated"? I thought you needed two different vectors to figure correlation? What is an example of correlated 1-D dataset; is it the "autocorrelation" mentioned re: Wall Street Returns?

   B. What does this assumption mean: "the errors $\epsilon_i$ for each observation are uncorrelated^_1 with common variance $\sigma^2$."? EDIT Maybe you can tell me if this assumption is the same as one of the 4 assumptions listed here Specifically I thought you needed two different vectors to figure correlation? Isn't $\epsilon_i$ like a vector of data, while $\sigma^2$ is a single scalar value?

2. Does this mean as $n$ approaches the size of the population, $SE(\hat{\mu})^2$ approaches zero? The Wikipedia article explicitly describes this tendency:

   ...the standard error of the mean of the sample will tend to zero with increasing sample size, because the estimate of the population mean will improve...

3. Author says of uncorrelated assumption, "This is clearly not true in Figure 3.1" -- but I don't see how the Figure 3.1 illustrates a correlation between errors $\epsilon_i$ and the common variance. Is it because, as they suggest in the caption "although it is somewhat deficient in the left of the plot."? How does this mathematically produce a correlation?

Answer 1

Edited Q3 answer to clarify.

Q1: yes, this is talking about bias/dependence in the observations/errors. Those formulae only hold strictly true if the data is IID (independent and identically distributed). If there is bias then you have to apply a correction.

Q2: yes, although the convergence will slow as you approach zero.

Q3: "somewhat deficient in the left side of the plot" is referring to the fact that the errors are imbalanced about the linear regression line (greater number below, greater values above). An ideal (or at least very good) regression line will have a fairly balanced distribution of errors above and below it. The imbalance is a visual illustration of what you asked about in Q1: that the errors in this data are not IID.

Answer 2

Q1

A: They could be correlated by a measure in the background. For instance, your sample might have been influenced by the time when you took it; the sun standing in a particular position or whatever. Then you would obtain another result than by sampling the whole day (what you might have intended).

B: As there is no precise definition of the symbol, we cannot say for sure. However, the reference to the plot implies what the author meant: The variance should be constant at every position $x_{i}$ -> TVs in the plot.

Q2

Clearly yes. Also the distribution of the mean gets closer to the normal distribution. This and this may help to understand.

Q3

I referred to this in my answer to Q1 B.