# Is the average of N linear regression coefficients equivalent to the regression coefficients for the entire dataset?

There have been some questions that are a bit similar to this one. However, I do not feel like the answers given truly answer the following question.

Suppose I have a large dataset of $$nN$$ points. I fit a simple linear regression model to it and obtain $$\hat{\beta}_0$$ and $$\hat{\beta}_1$$. Are these coefficients equivalent to the average of the $$N$$ coefficients obtained by fitting to the $$N$$ sub-samples of size $$n$$?

For this to be true I would need (taking $$\beta_1$$ as an example)

$$\beta_1 = \frac{\sum_{i=1}^{nN}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{nN}(x_i-\bar{x})^2}$$

to be identical to

$$\frac{1}{N}\sum_{k=1}^N\frac{\sum_{i=1}^{n}(x_i^{(k)}-\bar{x}^{(k)})(y_i^{(k)}-\bar{y}^{(k)})}{\sum_{i=1}^{n}(x_i^{(k)}-\bar{x}^{(k)})^2}$$

where $$k$$ refers to the individual partitions of size $$n$$.

I already suspect that they are not equivalent from numerical testing. If they are not equivalent, is it possible to say that one produces more accurate estimates than the other?

• why do you ask? you could combine the separate covariances to get the same coefficients Commented Oct 20, 2023 at 11:49
• I read that the estimates are unbiased and that their averages therefore tend the true coefficients. I was just wondering if there is a difference between the two ways of describing the coefficients. From the answers below, it seems they the rate at which they 'converge' is the same.
– hhh3
Commented Oct 20, 2023 at 15:00
• so using the averaged coefficients will give you a higher residual on the full data set than the regular LS estimate. Commented Oct 20, 2023 at 16:21

Let's call the two estimators $$\widehat{\beta}_1$$ (using the whole sample) and $$\widehat{\beta}_1^N$$ (the average across the $$N$$) subsamples. If the OLS assumptions are satified then these are both unbiased estimators for $$\beta_1$$, so they are equivalent in mean. If the equivalence you have in mind is equivalence in distribution, then again under OLS assumptions we have $$\frac{\widehat{\beta}_1-\beta_1}{se(\widehat{\beta}_1)}\overset{d}{\to}\mathcal{N}(0,1)$$ and this is also true for the second estimator as long as the dependence between estimators in each subsample is negliglible.

The two estimators are not identical, i.e. these are not alegbraically the same.

Addendum: Save we have non-random explanatory variables, i.e. $$x_i$$'s are deterministic. And say in this model, $$y_i=\beta_0+\beta_1 x_i + u_i$$, we have $$var(u_i)=\sigma_u^2$$. Then we have \begin{align} var(\hat{\beta}_1)&=\frac{\sigma_u^2}{\sum_{i=1}^{nN}(x_i-\bar x)^2}\\ & = \frac{\sigma_u^2}{\sum_{k=1}^{N}a_k}\\ var(\hat{\beta}_1^N)&=\frac1{N^2}\sum_{k=1}^N\frac{\sigma_u^2}{a_k}, \end{align} where $$a_k=\sum_{i=1}^N(x_i^{(k)}-\bar{x^{(k)}})^2$$. Now since $$\frac1N\sum\frac1{a_k}\geq\frac{1}{\frac1N\sum a_k}$$, $$\hat{\beta}_1$$ is more efficient compared to $$\beta_1^N$$. If $$x$$ is random, but has a constant variance and its relationship with $$u$$ is not varying across the sample (in terms of second moments), then the same result follows.

• and you would add that using all the data is more accurate? Commented Oct 20, 2023 at 11:46
• If you measure accuracy with "the asymptotic rate at which variance vanishes", then the you have a rate of $(nN)^{(-1)}$ for both estimators (under the above mentioned restrictions regarding the subsmaples). So they are asymptotically equivalent. Commented Oct 20, 2023 at 11:58
• how about that the ordinary one is BLUE en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem? Commented Oct 20, 2023 at 12:58
• The arguments I wrote were perched on an asymptotic understanding of the two estimators you provided, i.e. when n and N are both large, when the two estimators are not different. Gauss Markov assumes non-random regressors. Do you have non-random regressors in mind? Commented Oct 20, 2023 at 16:11
• @seanv507 ... the point with the asymptotic analysis is valid :-) - I added a bit more on the variance of the the estimators in a restricted setup. If we deviate from the simple framework of Gauss-Markov, then pinning down an ordering will require some more efforts. Commented Oct 20, 2023 at 17:38

I will expand the comment of seanv507 on the original post, sticking with the case of splitting the data into two pieces.

Let $$X$$ be the complete design matrix with $$n = n_1 + n_2$$ rows and $$p+1$$ columns, $$X_1$$ be the first $$n_1$$ rows and $$X_2$$ be the last $$n_2$$ rows. Similarly decompose the vector $$y$$ into $$y_1$$ and $$y_2$$.

The OLS regression of the complete data set is

\begin{align*} \hat{\beta} &= (X^\top X)^{-1} X^\top y\\ &= (X_1^\top X_1 + X_2^\top X_2)^{-1} (X_1^\top y_1 + X_2^\top y_2)\\ &= (X_1^\top X_1 + X_2^\top X_2)^{-1} (X_1^\top X_1 \hat{\beta}_1 + X_2^\top X_2 \hat{\beta}_2) \end{align*}

So you can exactly reconstruct the OLS estimate if you are willing from the covariance matrices $$X_1^\top X_1$$ and $$X_2^\top X_2$$. If you wanted to "update" your linear regression with new observations, instead of storing all $$n_1(p+1)$$ numbers in $$X_1$$ you only need to store the $$(p+1)^2$$ numbers in $$X_1^\top X_1$$

• Wont this basically grow to (p+1)^n with n folds/pieces, because you need the covariance across variables for each fold/piece? Commented Apr 25 at 21:13