Consider an i.i.d. sample $(X_1, Y_1), \dots, (X_N, Y_N)$, where each $X_i$ and $Y_i$ are $n$-dimensional column vectors, let $M \leq N$ and denote by $\hat{\beta}^{(M)}$ and $\hat{\beta}^{(N)}$ the coefficients resulting from applying linear regression to the sample comprising the first $M$/$N$ pairs, respectively. Is it true or false that $$ \hat{\beta}^{(M)}_i \sim \hat{\beta}^{(N)}_i $$ where $\hat{\beta}^{(K)}_i$ ($K\in\{M,N\}$) is the $i$th component of the vector $\hat{\beta}^{(K)}$.


1 Answer 1


Note that if $Y_i$'s are $n$-dimensional vectors, than $\hat\beta$ will be a matrix rather than a vector, if I understand your formulation correctly.

Never mind, the simple answer is no, the more samples, the more concentrated the value of $\hat\beta$ will be, so distributions will not be the same (I assume that's what $\hat\beta^{(M)} \sim \hat\beta^{(N)}$ means). You can try simulating an example:

import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt

N, M = 100, 10
b_N, b_M = [], []

for i in range(5000):
    x = np.random.random(N)
    y = 2*x + 1 + np.random.random(N)
    _, b_1 = sm.OLS(y, sm.add_constant(x)).fit().params
    _, b_1 = sm.OLS(y[:M], sm.add_constant(x[:M])).fit().params


different distributions

In theory it is possible to derive the distribution for $\hat\beta$ given the joint distribution for $(X, Y)$, arranging $n$ samples in matrices $\mathbf{X}, \mathbf{Y}$ and noting that $\hat\beta$ is a random variable $\hat\beta = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$.

I think that writing $\mathbf{X}^T \mathbf{X} = n \mathbf{Z}$ we can intuitively argue that increasing $n$ we get $\mathbf{Z}$ more and more concentrated around $E(X^T X)$ (and the same holds for $\mathbf{X^TY}$), so after inversion and cancellation of $n$ we are left with product of two random variables, which are more concentrated as $n$ grows.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.