# For linear least square regression, what is the relationship between the optimal solution and the empirical solution with finite samples

Consider a linear system $$y_i=Wx_i+v_i$$, where $$x_i\in R^{d\times1} \sim N(0,\Sigma_x)$$, $$v_i\in R^{p\times1} \sim N(0,\Sigma_v)$$, $$y_i\in R^{p\times1}$$ and $$W\in R^{p\times d}$$.

Now we consider a linear estimator $$\hat{x}_i=\beta_1y_i+\beta_0$$. We know that the optimal theoretical least square solution of is given by:

$$\beta^\ast_1=\Sigma_xW^\top(W\Sigma_xW^\top+\Sigma_v)^{-1}$$

$$\beta^\ast_0=0$$

On the other hand, suppose we have a set consisting of $$N$$ ovservations pairs $$\{(x_1,y_1),(x_2,y_2),..,(x_N,y_N)\}$$. We denote $$Y_N=[y_1,y_2,...,y_N]$$, $$X_N=[x_1,x_2,...,x_N]$$ and $$V_N=[v_1,v_2,...,v_N]$$. the abover system and the linear estimator can be written as

$$Y_N=WX_N+V_N$$

$$\hat{X}_N=[\beta_1, \beta_0]\begin{bmatrix}Y_N\\1\end{bmatrix}$$.

Therefore, the least square solution is given by :

$$[\beta^{\ast\ast}_1, \beta^{\ast\ast}_0]=X_N[Y^\top_N, 1^\top](\begin{bmatrix}Y_N\\1\end{bmatrix}[Y^\top_N, 1^\top])^{-1}$$.

where $$\beta^{\ast\ast}_1$$ and $$\beta^{\ast\ast}_0$$ can be considered as random variables.

We know that when $$N\to \inf$$, $$\beta^{\ast\ast}_1\to \beta^{\ast}_1$$.

My question is: when N is finite, can we describe the relationship between $$\beta^{\ast\ast}_1$$ and $$\beta^{\ast}_1$$, e.g., the pdf of $$\beta^{\ast\ast}_1-\beta^{\ast}_1$$, for a given $$N$$?