Let ($x_{i1}$, $x_{i2}$, ..., $x_{id}$, $y_i$), $i = 1,..., n$ be an i.i.d. multivariate sample and furthermore assume each observation belongs to one of possible $K$ categories. Assume for each category that the linear model holds and $E({\rm residual}\ i |x_i) = 0$ and ${\rm Var}({\rm residual}\ i | x_i) = \sigma^2$. Consider the two following methods to model the data:

i) Fit a separate linear model to each category.
ii) Fit a single interaction model to all the data.

Show that for a fixed category, methods (i) and (ii) will yield the same estimate of the regression line for that category.

My work thus far to prove i and ii are equivalent estimates of the regression line for category $A$ ($X_a$, $Y_a$, $B_a$).

To prove this I started by using: vector of Beta estimates for category $k = [B_k]$, matrix of estimates for category $k = [X_k]$, $Y$ estimates for category = $[Y_k]$.

  1. Find betas for the single category case from method i: $$[B_a] = (X_a' * X_a)^{-1} * (X_a' *Y_a)$$

  2. Simplifying method ii: Find betas for matrix of combined categories for the general case.

    In block diagonal form, Matrix $[X]$ becomes: \begin{bmatrix} X_a &0 &... &0 \\ 0 &X_b &... &0 \\ 0 &0 &... &X_k \end{bmatrix} Vector of Y's becomes:

\begin{bmatrix} Y_a \\ Y_b \\ ... \\ Y_k \end{bmatrix}

and you do the same multiplication $B = (X' * X)^{-1} * (X'*Y)$.

Multiplying this out I get: $$ (X' * X)^{-1} = \begin{bmatrix} 1/X_a^2 &0 &... &0 \\ 0 &1/X_b^2 &... &0 \\ 0 &0 &... &1/X_k^2 \end{bmatrix} $$

and, $$ X' * Y = \begin{bmatrix} X_aY_a &..... & &0 \\ 0 &X_bY_b &.... &0 \\ 0 &0 &... &X_kY_k \end{bmatrix} $$

multiplying these two results finally gives: $$ \vec{B} = \begin{bmatrix} B_a \\ B_b \\ ... \\ B_k\end{bmatrix} $$ so $B_a = Y_a * X_a'$.

Now that I'm this far I am not sure how to prove that method i yields $B_a = Y_a*X_a'$ without adding specific numbers. Does anyone know how to proceed? Any help would be appreciated.

  • $\begingroup$ Thank you for the work you did in transcribing the question (+1). One approach you might like to consider goes back to basics: consider how either of the two approaches might achieve a lower sum of squares of errors. It is intuitively obvious--and should take little effort to show--that neither approach can do better than the other in this respect. Assuming all parameters are identifiable, then each solution is unique--and therefore their fits must coincide. $\endgroup$ – whuber Dec 16 '14 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.