Let $Y_1, Y_2,Y_3$ be uncorrelated random variables with common variance $\sigma^2>0$ such that
$$ \begin{aligned} E[Y_1]&=\beta_1+\beta_2, \\ E[Y_2]&=2\beta_1 \\ E[Y_3]&=\beta_1-\beta_2 \end{aligned} $$ where $\beta_1$ and $\beta_2$ are unknown parameters. Find the residual (error) sum of squares under the above linear model. I proceed in this way:

Note that: $$ \begin{aligned} E[Y_1-\beta_1-\beta_2]&=0, \\ V[Y_1-\beta_1-\beta_2]&=\sigma^2, \\ E[Y_2-2\beta_1]&=0, \\ V[Y_2-2\beta_1]&=\sigma^2, \\ E[Y_3-\beta_1+\beta_2]&=0, \\ V[Y_3-\beta_1+\beta_2]&=\sigma^2 \end{aligned} $$ So the residual(error) sum of squares under the above linear model is: $$ \sum{e_i^2}=(Y_1-\beta_1-\beta_2)^2+(Y_2-2\beta_1)^2+(Y_3-\beta_1+\beta_2)^2 $$ Am I proceeding in the correct way? Can someone please explain what is meant by linear model here?

  • $\begingroup$ It seems correct to me. $\endgroup$
    – vinux
    Aug 2, 2012 at 13:18
  • $\begingroup$ To adapt my procdeure to your response, just replace estimators instead of parameters in your expression. The constant terms of these equations can be estimated by sample average. This can be shown for OLS estimators of a linear regression with a constant only. Thus $\hat \beta_1 + \hat \beta_2$ must be equal to Y1's mean. $2\hat\beta_1$ equals Y2'mean. This implies that in your third term, both betas may be expressed as Y1 and Y2 means. $\endgroup$
    – JDav
    Aug 2, 2012 at 20:58
  • $\begingroup$ I need more steps to proceed this problem $\endgroup$
    – Argha
    Aug 3, 2012 at 4:14

2 Answers 2


I don't remember encountering this problem before but mathematically you have correctly interpreted the assumptions. Now the next step is to estimate the 2 betas. How should that be done? All that is clear from the statement of the problem is that the error terms would be uncorrelated by implication and they would have the same variance. But it is not clear (1) whether or not they have the same distribution and (2) whether that distribution would be normal. If the error terms are normal they will be iid with 0 mean and common variance. You could then apply least squares to your formula you have for the error sum of squares. If the error terms are very non-normal but have the same distribution a robust ftting procedure like minimum sum of absolute deviations may be used. Keep in mind that uncorrelated residuals are not necessarily independent in non-normal situations.

This is called a linear model or more correctly a multivariate linear model because the response variables Y$_1$, Y$_2$ and Y$_3$ are all linear functions of the parameters.

  • $\begingroup$ I also stuck at that point.I know least square but can you explain robust ftting procedure of minimum sum of absolute deviations $\endgroup$
    – Argha
    Aug 2, 2012 at 14:00
  • $\begingroup$ @Rabnir You just minimize ∑ |e$_i$|= |Y$_1$-β$_1$-β$_2$|+|Y$_2-$2β$_1$|+|Y$_3$-β$_1$+β$_2$|. Other robust approach are possible also. $\endgroup$ Aug 2, 2012 at 14:24

Your model must be rewritten as an estimating equation of the form $Y=X\theta+\epsilon$ where $y_1,y_2,y_3$ are just the same observed variable at different groups (or clusters)

That implies: $y_i = \theta_1D1_i + \theta2 D2_i + \theta_3 D3_i + \epsilon_i$ Where $Dk_i$ is a dummy variable that =1 if the i-th observation corresponds to the $k-th$ group. So $\theta_1=\beta_1+\beta_2$ $\theta_2=2\beta_1$ $\theta_3=\beta_1-\beta_2$. But estimating the previous equations would yield 3 $\theta$ parameters from which only 2 $\beta$ exist so the system is overidentified. It would more reasonable to write $\theta_3$ as:

$\theta_3 = \theta_2-\theta_1$ thus the estimating equation becomes:

$y_i = \theta_1(D1_i-D3_i) + \theta2 (D2_i+D3_i) + \epsilon_i\equiv \theta_1x_{1i}+ \theta_2x_{2i} + \epsilon_i$

which is an standard linear regession with 2 explnatory variables. Once the model is estimated ($\hat \theta$), the sum of squared residuals is

$\sum_i\hat\epsilon_i^2 =\sum_i y_i^2-2 \sum_i y_i \hat y_i+ \sum \hat y_i^2$

The first term is obtained as $\sum (Y1_i + Y2_i +Y3_i)^2$. DEveloping the second term yields :

$-2(\hat\theta_1\sum_i y_i x_{1i}+\hat \theta_2 \sum y_i x_{2i}) $ $\equiv -2(\hat \theta_1 (\sum_i Y1_i-\sum Y3_i)+\hat \theta_2(\sum Y2_i- \sum Y3_i))$

while the third is:

$\hat\theta_1^2\sum_ix_{1i}^2 +\hat\theta_2^2\sum_ix_{2i}^2+2\hat\theta_1\hat\theta_2\sum_i{x_{1i}x_{2i}}$ where the sums are straigforward to calculate following the second term algebra.

But What about $\hat \theta$ values ? It's easy. Lets consider the first group observations, the equation becomes, $y_i=Y1_i= \theta_1 + \epsilon_i$, the second: $Y2_i=\theta_2 +\epsilon_i$ and third: $Y3_i=y_i=\theta_2-\theta_1+\epsilon_i$

These 3 equations are mutually exclusive as they are based on different samples so estimating the first and second (by OLS, ML etc) implies that the estimated $\theta$ are just averages of the corresponding $Yk$:

$\hat \theta_1 = \bar{Y1}$,and $\hat \theta_2=\bar{Y2}$ and therefore $\hat \theta_3=\bar Y2 -\bar{Y1}$

  • $\begingroup$ This strategy yield analytical expression for the summ of squared residuals as a function of Y1, Y2 and Y3 which are known. Im not sure if this is what you want. If this a software exercice , then the issue was only to write the estimating euqation properly as a function of 2 explanatory variables; calculating the residuals becomes trivial as it's always reported by the software. I forgot to rewrite the random term assumtions: $\epsilon_i$ iid, $V(\epsilon)=\sigma^2 $ for whatever $i$. $\endgroup$
    – JDav
    Aug 2, 2012 at 15:10
  • $\begingroup$ @J Dav: This is not a software problem and I explain what I know about this Problem $\endgroup$
    – Argha
    Aug 2, 2012 at 15:45
  • $\begingroup$ In that case the procedure is ok. As you can see the linear regression model : $Yk_i = \theta_1(D1_i - D3_i)+ \theta_2(D2_i+D3_i) +\epsilon_i $ satisfies the expectation equation that you wrote: $E(Yk_i) = \theta_1(D1_i - D3_i)+ \theta_2(D2_i+D3_i) $ To get $E(Y1)$ just assume $D1_i=1$ while the others are 0. The same for the others. If you require further explnations please do not hesitate to ask. $\endgroup$
    – JDav
    Aug 2, 2012 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.