Is the average of N linear regression coefficients equivalent to the regression coefficients for the entire dataset?

Question

There have been some questions that are a bit similar to this one. However, I do not feel like the answers given truly answer the following question.

Suppose I have a large dataset of $nN$ points. I fit a simple linear regression model to it and obtain $\hat{\beta}_0$ and $\hat{\beta}_1$. Are these coefficients equivalent to the average of the $N$ coefficients obtained by fitting to the $N$ sub-samples of size $n$?

For this to be true I would need (taking $\beta_1$ as an example)

$$ \beta_1 = \frac{\sum_{i=1}^{nN}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{nN}(x_i-\bar{x})^2} $$

to be identical to

$$ \frac{1}{N}\sum_{k=1}^N\frac{\sum_{i=1}^{n}(x_i^{(k)}-\bar{x}^{(k)})(y_i^{(k)}-\bar{y}^{(k)})}{\sum_{i=1}^{n}(x_i^{(k)}-\bar{x}^{(k)})^2} $$

where $k$ refers to the individual partitions of size $n$.

I already suspect that they are not equivalent from numerical testing. If they are not equivalent, is it possible to say that one produces more accurate estimates than the other?

Answer 1

Let's call the two estimators $\widehat{\beta}_1$ (using the whole sample) and $\widehat{\beta}_1^N$ (the average across the $N$) subsamples. If the OLS assumptions are satified then these are both unbiased estimators for $\beta_1$, so they are equivalent in mean. If the equivalence you have in mind is equivalence in distribution, then again under OLS assumptions we have $\frac{\widehat{\beta}_1-\beta_1}{se(\widehat{\beta}_1)}\overset{d}{\to}\mathcal{N}(0,1)$ and this is also true for the second estimator as long as the dependence between estimators in each subsample is negliglible.

The two estimators are not identical, i.e. these are not alegbraically the same.

Addendum: Save we have non-random explanatory variables, i.e. $x_i$'s are deterministic. And say in this model, $y_i=\beta_0+\beta_1 x_i + u_i$, we have $var(u_i)=\sigma_u^2$. Then we have \begin{align} var(\hat{\beta}_1)&=\frac{\sigma_u^2}{\sum_{i=1}^{nN}(x_i-\bar x)^2}\\ & = \frac{\sigma_u^2}{\sum_{k=1}^{N}a_k}\\ var(\hat{\beta}_1^N)&=\frac1{N^2}\sum_{k=1}^N\frac{\sigma_u^2}{a_k}, \end{align} where $a_k=\sum_{i=1}^N(x_i^{(k)}-\bar{x^{(k)}})^2$. Now since $\frac1N\sum\frac1{a_k}\geq\frac{1}{\frac1N\sum a_k}$, $\hat{\beta}_1$ is more efficient compared to $\beta_1^N$. If $x$ is random, but has a constant variance and its relationship with $u$ is not varying across the sample (in terms of second moments), then the same result follows.

Answer 2

I will expand the comment of seanv507 on the original post, sticking with the case of splitting the data into two pieces.

Let $X$ be the complete design matrix with $n = n_1 + n_2$ rows and $p+1$ columns, $X_1$ be the first $n_1$ rows and $X_2$ be the last $n_2$ rows. Similarly decompose the vector $y$ into $y_1$ and $y_2$.

The OLS regression of the complete data set is

$$ \begin{align*} \hat{\beta} &= (X^\top X)^{-1} X^\top y\\ &= (X_1^\top X_1 + X_2^\top X_2)^{-1} (X_1^\top y_1 + X_2^\top y_2)\\ &= (X_1^\top X_1 + X_2^\top X_2)^{-1} (X_1^\top X_1 \hat{\beta}_1 + X_2^\top X_2 \hat{\beta}_2) \end{align*} $$

So you can exactly reconstruct the OLS estimate if you are willing from the covariance matrices $X_1^\top X_1$ and $X_2^\top X_2$. If you wanted to "update" your linear regression with new observations, instead of storing all $n_1(p+1)$ numbers in $X_1$ you only need to store the $(p+1)^2$ numbers in $X_1^\top X_1$