# Simultaneous estimation of a group of linear model (regression) parameters

Suppose $$y=ax+z$$ where $$x, y, z$$ are random variables with range in $$\mathbf R$$, $$\mathbf E[x]=\mathbf E[z|x]=0$$ and $$a$$ is a constant. Note the distribution of $$z$$ conditioned on $$x$$ depends on $$x$$. This distribution is unknown. Suppose $$(x_j,y_{i,j})$$ is a tuple of sample observation of $$(x,y)$$ where $$y_{i,j}=a_ix_{i,j}+z_{i,j}$$ where $$(i,j)\in I$$ for some (finite) index set $$I$$. How do we estimate $$\{a_i\}$$?

We can of course use a separate linear regression on the set $$O_i:=\{(x_{i,j},y_{i,j})|h=i,(h,j)\in I\}$$ for each given $$i\in\{i|(i,j)\in I\}$$.

1) Is there a way to have a better estimate of $$\{a_i\}$$ by considering the sample observations $$\{(x_{i,j},y_{i,j})\}_{(i,j)\in I}$$ as a whole ensemble simultaneously?

2) We can estimate the distribution of $$z$$ conditioned on $$x$$ by pooling the residue of the ordinary linear regression on each $$i$$. Is there perhaps an iterative procedure to improve the estimation of the coefficient set $$\{a_i\}$$?

3) Seemingly unrelated regression(SUR) is suggested by Jesper Hybel in his comment below as a way to treat this problem. However, SUR requires $$\{z_{i,j}\}_i$$ for a given $$j$$ to be drawn from the same distribution dependent on $$j$$ (e.g. time) to estimate the covariance. In my setting, $$(i,j)$$ is only a sampling label. The probability density of $$z$$ conditioned on $$x$$ depends on $$x$$ and not on the labeling, which can be permuted arbitrarily. $$x_{i,j}$$'s may all be distinct.

• I do not understand what you mean by "separate linear regression on the set" $I_i$. The way you have specified this set it seems to me that it is the entire set of observations. Did you mean $I_i = \{(x_j,y_{hj})\lvert h = i, (h,j) \in I\}$? – Jesper Hybel Dec 20 '18 at 12:08
• @JesperHybel: Yes, good catch. I have corrected it. Thank you. Do you have any idea regarding the question? – Hans Dec 20 '18 at 18:44
• Well something bugs me. I notice that $y_{1j} - y_{2j} = (a_1-a_2)x_j$ so $(y_{1j} - y_{2j})/x_j = (a_1-a_2)$ take then another $j$ lets call it $j'$ then similarly for same pair of $i$'s you get $(y_{1j'} - y_{2j'})/x_{j'} = (a_1-a_2)$ are both these expressions really satisfied in the data? Are there perhaps missing an index on $z$ so it should be $z_{ij}$ not just $z_j$? – Jesper Hybel Dec 20 '18 at 19:17
• @JesperHybel: You are again correct. Thank you. I was sloppy regarding the notations. I have now corrected them accordingly. Please review. – Hans Dec 20 '18 at 20:03
• starting to look like a SUR model to me. Have a look at sur – Jesper Hybel Dec 20 '18 at 20:11