How to interpret confidence interval and prediction interval in simple regression "in/with the context of sampling distribution"?

Question

With the context of sampling distribution, in regression analysis, is the following an appropriate interpretation?

Assumptions :

1. X & Y have a linear relationship
2. sample size is large enough for Central Limit Theorem to be applicable

Sampling Distribution of the Mean of Y
If all possible samples, m, of sample size n were taken for a specific value of X (i.e. X = h), and mean ($\mu_{y}$) calculated for the Ys observed against those Xs, then :

1. The distribution of such means will be normally distributed.
2. 95% of the confidence intervals ($ \mu_{y} \: \pm \: 1.96 \sigma$) calculated from such estimated means will contain the true population mean of Y.
3. Mean of all the m means will be equal to the overall population mean of Y. (unbiased estimator of a population parameter)
   (A statistic is called an unbiased estimator of a population parameter if the mean of the sampling distribution of the statistic is equal to the value of the parameter.)

But since its not practical to take all possible samples of size n, we take 1 sample of size n, and calculate mean of y.

Confidence Interval for Y
We expect that, $\mu_{y}$ obtained from this one sample is one of those 95% that will yield a confidence interval that will contain the true population mean of Y.

Prediction Interval for Y
The 95% prediction interval (that has a much wider reach than the confidence interval) calculated using the $\mu_{y}$ obtained from this one sample will contain 95% of all the possible population values of Y.

Have I got the interpretation for CI & PI right ?

Answer 1

Technically, here is the definition of a finite sampling distribution, for example, per Wikipedia, to quote:

In statistics, a sampling distribution or finite-sample distribution is the probability distribution of a given random-sample-based statistic. If an arbitrarily large number of samples, each involving multiple observations (data points), were separately used in order to compute one value of a statistic (such as, for example, the sample mean or sample variance) for each sample, then the sampling distribution is the probability distribution of the values that the statistic takes on.

As such, assume we are interested in the distribution of values of say slope in a regression analysis, then one could enumerate, for all pairs of two selected points, the implied slope value of a line based on these determining points.

Such a measure exists in robust regression analysis. As an example, consider the Theil-Sen estimator, to quote Wikipedia on this robust technique:

In non-parametric statistics, the Theil–Sen estimator is a method for robustly fitting a line to sample points in the plane (simple linear regression) by choosing the median of the slopes of all lines through pairs of points. It has also been called Sen's slope estimator,[1][2] slope selection,[3][4] the single median method,[5] the Kendall robust line.[7]

And, further relating to CI construction:

A confidence interval for the slope estimate may be determined as the interval containing the middle 95% of the slopes of lines determined by pairs of points[12] and may be estimated quickly by sampling pairs of points and determining the 95% interval of the sampled slopes. According to simulations, approximately 600 sample pairs are sufficient to determine an accurate confidence interval.[10]

My answer may provide you with some more background on the general application to regression analysis of a finite sampling methodology.

Answer 2

The interpretation of confidence intervals and prediction intervals can be tricky in a linear regression modelling setting for novice learners since it requires then to distinguish between what happens in practice and what happens in theory.

To illustrate this distinction, let't imagine the following scenario:

We are interested in studying the relationship between height (X) and weight (Y) for students at a local university. In particular, we would like to use this relationship to estimate the average weight in kg of all students with a height of 1.60 metres.

In practice, we will estimate the average weight of interest by selecting a random sample of 100 students, say, from all students enrolled at the university and measuring their heights and weights. We will then plug these heights and weights into the appropriate formulas for constructing the confidence interval we are interested in. This will yield a 95% confidence interval for the average weight of all students at that university of (60kg, 70kg), say. We will interpret this interval by saying something like: We are 95% confident that the average weight of all students at this local university who have a height of 1.60 metres ranges between 60kg and 70kg.

The reason we are able to make this last statement is because we are using a statistical procedure for constructing the 95% confidence intervals which has good properties in the long run. This is where the theory comes in:

If we were to perform a thought experiment and repeat what we did in practice many, many times - but each time, drawing a different random sample of size 100 from the list of students enrolled at the university - we would obtain a very, very large collection of 95% confidence intervals for the average weight of all students of the university with a height of 1.60 metres. These intervals might look like: (62kg, 73kg), (59kg, 68kg), etc., so they would have different endpoints.

In our thought experiment, we could assume for a moment we are God - this means we would know what the true value of the quantity we are after actually is. For example, the quantity we are trying to estimate would be known to God to be 65kg - this quantity would represent the average weight of all students at the university who have a height of 1.60 metres.

If we were to examine the 95% confidence intervals in our thought experiment one by one to see whether or not they contain the true value we are interested in, we would expect according to the theory governing the statistical procedure we used to construct these intervals, that 95% of them would include this true value of 65kg.

It is because of this theoretical expectation that we could make our practical statement:

We are 95% confident that the average weight of all students at this local university who have a height of 1.60 metres ranges between 60kg and 70kg.

To be continued...