Standard Error, Standard Deviation and Variance confusion

Question

I am quite confused in these terminologies (especially but not limited to regression)
I do understand what Variance and Standard Deviation means, they measure the dispersion / variability of the data.

However, according to my understanding, Standard Error $= \frac{s}{\sqrt{n}}$ where s is the sample standard deviation.
But in regression (for simplicity, here refer to Simple Linear Regression but MLR shall be of the same fashion) $y = \beta_0 + \beta_{1}x + e$.
Variance of $\hat\beta_1$ = $\frac{\sigma^2}{SXX}$
And while we are doing confidence interval for $\hat\beta_1$, the SE we use is simply the square root of Var($\hat\beta_1$) = $\frac{\sigma}{\sqrt{SXX}}$ without needing to divide by $\sqrt{n}$

My questions:
1) Is there a difference between normal Standard Error (of mean) that we talk about (i.e. $\frac{s}{\sqrt{n}}$) and the SE we talk in regression.
2) I suppose, $\hat\beta_1$ is not a mean but purely an estimator of the parameter $\beta_1$, so why do we use SE when we are constructing confidence interval of $\hat\beta_1$?
3) What about confidence interval for predicted $y$ value and fitted $y$ value respectively?

Answer 1

The term "standard error" refers to the standard deviation of a statistic that is calculated. So, you can calculate a standard error for a mean--because the mean is a statistic. You can also calculate a standard error for a parameter estimate like $\hat{\beta}$.

We say standard error instead of standard deviation to distinguish between a value that's calculated from repeated observations and an estimate that's based on a theory about the distribution.

We only have one observation for $\hat{\beta}$, and we have mathematical theory to derive its sampling error--so we call that the standard error.

We have more than one observation of a variable X, and we calculate the sampling error based on that observed data--so we call that statistic the standard deviation.

Answer 2

The terminology is the same everywhere in statistics I think:

• Variance $\sigma^2$ is the second moment of a known probability distribution
• Standard Deviation $\sigma$ is the square root of variance
• Variance of the mean $\sigma^2_{\mu} = \frac{\sigma^2}{N}$ is the variance of the mean of $N$ i.i.d random variables
• Standard Deviation of the Mean $\sigma_{\mu}$ is the square root of the variance of the mean

The 4 above metrics apply analytically to probability distributions. One can estimate any one of them, typically denoted by letter $s$ and prefix 'sample', such as 'sample error of the mean' $s_{\mu}$. Sample standard deviation and Sample standard deviation of the mean are also known as Standard Error and Standard Error of the mean (SEM) respectively

With respect to your questions:

• Variance and standard deviation are metrics of the distribution of the random variables in analytic case and a metric of data in the sample case. These terms are not applicable to parameters of your model, such as $\beta$ or $\hat \beta$. These are simply the parameter and its estimate.
• When you construct a confidence interval for an unknown parameter, you perform a hypothesis test. The confidence interval is likely to be a function of the moments of the distribution, or their sample counterparts, but that depends strongly on the underlying distribution.
• Confidence intervals only apply to unknown parameters of the model, they do not apply to parts of data such as $y$. The closest entity to a confidence interval when applied to random variable itself is a tolerance interval, namely, the interval where the random variable is likely to fall given the exact model parameters