6
$\begingroup$

Quoting from An Introduction to Statistical Learning with Applications in R (James, Witten, Hastie, Tibshirani), Chapter 3 on Linear Regression. I used numbered superscripts _1, _2, _3 to mark the areas where I have a corresponding question:

We have established that the average of $\hat{\mu}$ over many data sets will be very close to $\mu$ , but that a single estimate $\hat{\mu}$ may be a substantial underestimate or overestimate of $\mu$. How far off will that single estimate of $\hat{\mu}$ be? In general, we answer this question by computing the standard error of $\hat{\mu}$, written as $SE(\hat{\mu})$. We have the well-known formula:

$Var(\hat{\mu}) = SE(\hat{\mu})^2 = \frac{\sigma^2}{n}$

where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$ (This formula holds provided that the $n$ observations are uncorrelated_1.) Roughly speaking, the standard error tells us the average amount that this estimate $\hat{\mu}$ differs from the actual value of $\mu$. Equation 3.7 also tells us how this deviation shrinks with $n$—the more observations we have, the smaller the standard error of $\hat{\mu}$_2. In a similar vein, we can wonder how close $\hat{\beta_0}$ and $\hat{\beta_1}$ are to the true values $\beta_0$ and $\beta_1$. To compute the standard errors associated with $\hat{\beta_0}$ and $\hat{\beta_1}$, we use the following formulas:

$SE(\hat{\beta_0})^2 = \sigma^2\left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i-\bar{x})^2}\right]$

$SE(\hat{\beta_1})^2 = \frac{\sigma^2}{\sum_{i=1}^n (x_i-\bar{x})^2}$

where $\sigma^2 = Var(\epsilon)$. For these formulas to be strictly valid, we need to assume that the errors $\epsilon_i$ for each observation are uncorrelated_1 with common variance $\sigma^2$. This is clearly not true in Figure 3.1_3, but the formula still turns out to be a good approximation.

For your reference, here is figure 3.1: least squares fit on advertising dollars spent on TV (x-axis) vs Sales (y-axis)

Questions and attempts at answers:

  1. I've marked two areas the assumption of "uncorrelated" data appears.

    A. What does it mean for a 1-dimensional dataset to be "correlated"? I thought you needed two different vectors to figure correlation? What is an example of correlated 1-D dataset; is it the "autocorrelation" mentioned re: Wall Street Returns?

    B. What does this assumption mean: "the errors $\epsilon_i$ for each observation are uncorrelated_1 with common variance $\sigma^2$."? EDIT Maybe you can tell me if this assumption is the same as one of the 4 assumptions listed here Specifically I thought you needed two different vectors to figure correlation? Isn't $\epsilon_i$ like a vector of data, while $\sigma^2$ is a single scalar value?

  2. Does this mean as $n$ approaches the size of the population, $SE(\hat{\mu})^2$ approaches zero? The Wikipedia article explicitly describes this tendency:

    ...the standard error of the mean of the sample will tend to zero with increasing sample size, because the estimate of the population mean will improve...

  3. Author says of uncorrelated assumption, "This is clearly not true in Figure 3.1" -- but I don't see how the Figure 3.1 illustrates a correlation between errors $\epsilon_i$ and the common variance. Is it because, as they suggest in the caption "although it is somewhat deficient in the left of the plot."? How does this mathematically produce a correlation?
$\endgroup$
3
$\begingroup$

Edited Q3 answer to clarify.

Q1: yes, this is talking about bias/dependence in the observations/errors. Those formulae only hold strictly true if the data is IID (independent and identically distributed). If there is bias then you have to apply a correction.

Q2: yes, although the convergence will slow as you approach zero.

Q3: "somewhat deficient in the left side of the plot" is referring to the fact that the errors are imbalanced about the linear regression line (greater number below, greater values above). An ideal (or at least very good) regression line will have a fairly balanced distribution of errors above and below it. The imbalance is a visual illustration of what you asked about in Q1: that the errors in this data are not IID.

$\endgroup$
  • 1
    $\begingroup$ "referring to the fact that the errors are imbalanced about the linear regression line (greater number below, greater values above)." Do you mean as the value of the x-axis increases, the errors get further from the regression line? When x = 50 they are within about +/- 2.5, but when x=250 the errors are +/- 7? $\endgroup$ – The Red Pea Aug 23 '16 at 13:59
  • 1
    $\begingroup$ Not exactly. If you look at the right side of the plot you can see that the regression line bisects the errors quite neatly (even distribution of them above and below the line). Whereas on the left the errors look like they've been 'shifted' upwards. You would expect the size of the errors to increase as you move away from the mean of x, but if they are 'moving' in one way or the other that's a sign that the errors are not IID/uncorrelated. $\endgroup$ – hamedbh Aug 23 '16 at 14:07
  • $\begingroup$ OK, your explanation was my first interpretation. I see that when x is beneath/ to the left ~20 all the errors fall below the regression line (rather than evenly above and below). Hence "deficient to the left". But how does this translate to "correlation between errors $\epsilon_i$ and the common variance"? $\epsilon_i$ is the height of those lines, but how do I visualize common variance in terms of that graph? And how can $\epsilon_i$ (a vector of values?) correlate with common variance (a single value?) $\endgroup$ – The Red Pea Aug 23 '16 at 14:21
  • $\begingroup$ I think the author is pointing out two assumptions, and I misread... "the errors $\epsilon_i$ must be uncorrelated" -- that is (I) "statistical independence of the errors"-- and "with common variance $\sigma^2$" -- that is (II) "homoscedasticitiy (constant variance) of the errors". The former is the effect you describe, which seems similar to "mean of zero" effect (deficient on the left); homoscedasticity (common variance) is what I describe, which is observed as x moves to the right. $\endgroup$ – The Red Pea Aug 24 '16 at 11:53
  • $\begingroup$ I think you're right. $\endgroup$ – hamedbh Aug 24 '16 at 11:55
1
$\begingroup$

Q1

A: They could be correlated by a measure in the background. For instance, your sample might have been influenced by the time when you took it; the sun standing in a particular position or whatever. Then you would obtain another result than by sampling the whole day (what you might have intended).

B: As there is no precise definition of the symbol, we cannot say for sure. However, the reference to the plot implies what the author meant: The variance should be constant at every position $x_{i}$ -> TVs in the plot.

Q2

Clearly yes. Also the distribution of the mean gets closer to the normal distribution. This and this may help to understand.

Q3

I referred to this in my answer to Q1 B.

$\endgroup$
  • $\begingroup$ " As there is no precise definition of the symbol, " which symbol? My quote includes the author's definition of $\sigma^2=Var(\epsilon)$... Oh! You mean there is no definition of $\epsilon$ itself? i.e. we can't know it? I agree... But what do you mean "The variance should be constant at every position $x_i$ -> TVs in the plot."? Where is it not constant? , When $x_i$ < 50? $\endgroup$ – The Red Pea Aug 23 '16 at 13:16
  • 1
    $\begingroup$ The variance in the number of sales appears to be less for small values of TV for instance. You can determine a growing variance by the increasing spread in the values for sales as TV increases. $\endgroup$ – Jan Rothkegel Aug 24 '16 at 19:17
  • $\begingroup$ OK, great, so as I posted to the other answerer; the author was pointing out two assumptions. @hamedbh identified the one (statistical independence), and you identified the other (homoscedasticity) in your Q1)B). Thank you. I would accept the answer, but I feel like I want to combine Jan and hamedbh's answer. $\endgroup$ – The Red Pea Aug 24 '16 at 22:04
  • $\begingroup$ OK once I got Java to work, those links you posted about Central Limit Theorem are interesting , but can you tell me how they relate to the concept of "Standard Error"? Neither site uses those words -- (I specifically interpret "standard deviation" --which is mentioned -- as something different than "standard error"...) $\endgroup$ – The Red Pea Aug 25 '16 at 14:33
  • $\begingroup$ The standard error is the standard deviation of the distribution of the mean. That is, the sample mean follows a normal distribution and with increasing sample size the spread reduces as the square root of the sample size increases. The standard deviation of such particular distribution of the mean is called the standard error. $\endgroup$ – Jan Rothkegel Aug 26 '16 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.