Suppose $y=ax+z$ where $x, y, z$ are random variables with range in $\mathbf R$, $\mathbf E[x]=\mathbf E[z|x]=0$ and $a$ is a constant. Note the distribution of $z$ conditioned on $x$ depends on $x$. This distribution is unknown. Suppose $(x_j,y_{i,j})$ is a tuple of sample observation of $(x,y)$ where $y_{i,j}=a_ix_{i,j}+z_{i,j}$ where $(i,j)\in I$ for some (finite) index set $I$. How do we estimate $\{a_i\}$?

We can of course use a separate linear regression on the set $O_i:=\{(x_{i,j},y_{i,j})|h=i,(h,j)\in I\}$ for each given $i\in\{i|(i,j)\in I\}$.

1) Is there a way to have a better estimate of $\{a_i\}$ by considering the sample observations $\{(x_{i,j},y_{i,j})\}_{(i,j)\in I}$ as a whole ensemble simultaneously?

2) We can estimate the distribution of $z$ conditioned on $x$ by pooling the residue of the ordinary linear regression on each $i$. Is there perhaps an iterative procedure to improve the estimation of the coefficient set $\{a_i\}$?

3) Seemingly unrelated regression(SUR) is suggested by Jesper Hybel in his comment below as a way to treat this problem. However, SUR requires $\{z_{i,j}\}_i$ for a given $j$ to be drawn from the same distribution dependent on $j$ (e.g. time) to estimate the covariance. In my setting, $(i,j)$ is only a sampling label. The probability density of $z$ conditioned on $x$ depends on $x$ and not on the labeling, which can be permuted arbitrarily. $x_{i,j}$'s may all be distinct.

  • $\begingroup$ I do not understand what you mean by "separate linear regression on the set" $I_i$. The way you have specified this set it seems to me that it is the entire set of observations. Did you mean $I_i = \{(x_j,y_{hj})\lvert h = i, (h,j) \in I\}$? $\endgroup$ – Jesper Hybel Dec 20 '18 at 12:08
  • $\begingroup$ @JesperHybel: Yes, good catch. I have corrected it. Thank you. Do you have any idea regarding the question? $\endgroup$ – Hans Dec 20 '18 at 18:44
  • $\begingroup$ Well something bugs me. I notice that $y_{1j} - y_{2j} = (a_1-a_2)x_j$ so $(y_{1j} - y_{2j})/x_j = (a_1-a_2)$ take then another $j$ lets call it $j'$ then similarly for same pair of $i$'s you get $(y_{1j'} - y_{2j'})/x_{j'} = (a_1-a_2)$ are both these expressions really satisfied in the data? Are there perhaps missing an index on $z$ so it should be $z_{ij}$ not just $z_j$? $\endgroup$ – Jesper Hybel Dec 20 '18 at 19:17
  • $\begingroup$ @JesperHybel: You are again correct. Thank you. I was sloppy regarding the notations. I have now corrected them accordingly. Please review. $\endgroup$ – Hans Dec 20 '18 at 20:03
  • 1
    $\begingroup$ starting to look like a SUR model to me. Have a look at sur $\endgroup$ – Jesper Hybel Dec 20 '18 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.