Minimize $\mathrm{E}(U-bV-a)^{2}$, $U=\sum_{i=1}^{3}X_{i}.\, V=\sum_{i=1}^{3}i X_{i}\quad X_{1},X_{2},X_{3}\in N(1,1)$ i.i.d. r.v

Question

Let $X_{1}, X_{2}$ and $X_{3}$ be independent , $N(1,1)$-distributed random variables. Set $U=X_{1}+X_{2}+X_{3}$ and $V=X_{1}+2X_{2}+3X_{3}$. Determine the constants $a$ and $b$ so that $\mathrm{E}(U-a-bV)^{2}$ is minimized.

Here is what I did

\begin{equation*} \begin{split} U-bV\in N(B\mu,B\Lambda B^{T})\\ \mathrm{E}((U-bV)-a)^{2}\\ \frac{d}{d a} \mathrm{E}((U-bV)-a)^{2}=-2\mathrm{E}(U-bV)+2a=2(a-\mathrm{E}(U-bV))\\ \mu= \left[\begin{array}{c} U \\ bV \end{array}\right] \left[\begin{array}{ccc} 1&1& 1 \\ b&2b&3b \end{array}\right]\left[\begin{array}{c} 1 \\ 1\\ 1 \end{array}\right]=\left[\begin{array}{c} 3 \\ 6b \end{array}\right]\\ B\mu=\left[\begin{array}{cc} 1 & -1 \end{array}\right]\left[\begin{array}{c} 3 \\ 6b \end{array}\right]=\left[3-6b\right]\\ 2(a-\mathrm{E}(U-bV))=2(3-6b-a) \end{split} \end{equation*}

My book gives the answer

\begin{equation*} b=a=\frac{3}{7}\Leftrightarrow 2(3-6b-a)=0 \end{equation*} which of course works. But is there any reason that this is the only solution? Is there any reason for example

\begin{equation*} b=\frac{1}{3},\, a=1\quad \text{or}\quad b=-1,\, a=9 \end{equation*}

wouldn't work as well?

Answer 1

You can multiply out $\mathrm{E}(U-a-bV)^{2} $ keeping $ U $ and $ V $. Differentiate wrt $ a $ to get an equation involving $ a $ and $ b $ in terms of expectations of $ U $ and $ V $. Do the same for $ b $. You then have two simultaneous equations in $ a $ and $ b $.

Solve for $ a $ and $ b$. You should get..

$ \hat b = Cov (U,V)/Var(V) $

$ \hat a = E(U)-\hat bE(V) $

This agrees with the usual least squares result.

Next, plug in the values of $ U $ and $ V $ in terms of the $ X_i $.

$ Cov (X_i, X_j) = 0 $ if $ i \ne j $. $ Cov (X_i, X_i) = Var(X_i) $

Answer 2

\begin{equation*} \begin{split} U,V\in N(\mu,\Lambda)\\ \mu=\left[\begin{array}{ccc} 1&1& 1 \\ 1&2&3 \end{array}\right]\left[\begin{array}{c} 1 \\ 1\\ 1 \end{array}\right]=\left[\begin{array}{c} 3 \\ 6 \end{array}\right]\\ \Lambda= \left[\begin{array}{ccc} 1 & 1& 1 \\ 1 & 2& 3 \end{array}\right]\left[\begin{array}{cc} 1 &1 \\ 1 & 2\\ 1& 3 \end{array}\right]=\left[\begin{array}{cc} 3 & 6\\ 6 & 14 \end{array}\right]\\ \mathrm{E}((U-a)-bV)^{2}=\mathrm{E}\Big[(U-a)^{2}-2b(U-a)V+b^{2}V^{2}\Big]\\ =\mathrm{E}\Big[U^{2}-2aU+a^{2}-2bUV+2baV+b^{2}V^{2}\Big]\\ =\mathrm{E}(U^{2})-2a\mathrm{E}(U)+a^{2}-2b\mathrm{E}(UV)+2ba\mathrm{E}(V)+b^{2}\mathrm{E}(V^{2})\\ \mathrm{E}(U)=3,\quad \mathrm{E}(V)=6,\qquad \mathrm{Var}(U)=\mathrm{E}(U^{2})-(\mathrm{E}(U))^{2}\\ \mathrm{Var}(V)=\mathrm{E}(V^{2})-(\mathrm{E}(V))^{2}\\ \mathrm{E}(U^{2})=3+(3)^{2}=12\\ \mathrm{Var}(V^{2})=14+(6)^{2}=50\\ \mathrm{Cov}(U,V)=\mathrm{E}(UV)-\mathrm{E}(U)\mathrm{E}(V)\\ \mathrm{E}(UV)=6+3\cdot 6=24\\ \mathrm{E}((U-a)-bV)^{2}=12-6a+a^{2}-48b+12ba+50b^{2}\\ \frac{d}{da}\mathrm{E}((U-a)-bV)^{2}=0=-6+2a+12b\\ a=\frac{-6+12b}{-2}=3-6b\\ \frac{d}{db}\mathrm{E}((U-a)-bV)^{2}=0=-48+12a+100b\\ =-48+12(3-6b)+100b\\ =-12+28b\\ \frac{3}{7}=b\\ a=3-6(\frac{3}{7})=\frac{3\cdot 7-3\cdot 6}{7}=\frac{3}{7} \end{split} \end{equation*}