1
$\begingroup$

I came across this mle formula in some (undocumented) code performing linear regression with input matrices $A$ and $B,$ and was wondering how it was derived. It might also be some level of approximation, I'm not entirely sure.

$y$ is a $N$-dimensional vector, $A$ is $N \times M$ matrix, $B$ is $N \times P$ matrix

$$y \sim \operatorname{Normal}(A\alpha + B\beta, \sigma^2I)$$

the MLE $\hat{\alpha}$ is supposedly $\hat{\alpha} = \frac{y \cdot A \: - \: (Q^TA \:\cdot\: Q^Ty)}{A A \: - \: (Q^TA \:\cdot \:Q^TA)}$

where $Q$ in the above formula is $Q$ from QR-decomposition on $B$ (the $Q$ in this case is from thin/skinny QR-decomp, but I doubt it affects the derivation). I don't understand where $\hat{\alpha}$ formula comes from, any help would be appreciated!

Improve this question
$\endgroup$
4
  • $\begingroup$ A, B and y are defined, but X is not. ... What's X? What matrix is Q from the QR decomposition of? Normally, the parameters of a normal are mean and variance. $y$ is presumably an N-vector but you appear to have a scalar variable to represent the variance-covariance matrix (additionally lower-case $\sigma$ is normally used to stand for standard deviation). Do you intend to have $\sigma^2 I$ there or something else? Is there any information about what this code is for? $\endgroup$
    – Glen_b
    Commented Jul 7, 2019 at 9:12
  • $\begingroup$ X was me getting my variables confused, and I further clarified the question. If I am still missing information let me know, and I will try my best to add it. Thanks! $\endgroup$
    – user252944
    Commented Jul 7, 2019 at 20:31
  • $\begingroup$ What does $\text{“ }A\cdot y\text{ ''}$ mean when $A$ is an $N\times M$ matrix and $y$ is an $N\times 1$ matrix? $\endgroup$
    – Michael Hardy
    Commented Jul 7, 2019 at 23:52
  • $\begingroup$ I imagine y is 1 x N, and I just changed it to y⋅A $\endgroup$
    – user252944
    Commented Jul 8, 2019 at 0:42

1 Answer 1

Reset to default
2
$\begingroup$

Intuitively, our answer must consider some projection of the data onto the residual from the prediction of B: Q provides an orthonormal basis for B, so we must remove the components on Q. More rigorously, We can directly compute the MLE using the formula for linear estimator. We have that $$ \begin{bmatrix} A & B \end{bmatrix} \begin{bmatrix} \alpha \\ \beta \end{bmatrix} + \epsilon = y $$ Using the formula to solve for $\hat{\alpha}, \hat{\beta}$, we must have $$ \begin{bmatrix} A^T A & A^T B \\ B^T A && B^T B \end{bmatrix}^{-1} \begin{bmatrix} A \\ B \end{bmatrix} y = \begin{bmatrix} \hat{\alpha} \\ \hat{\beta} \end{bmatrix} $$ Using the formula for 2x2 block matrix inversion, we have $$ \hat{\alpha} = ((A^T A - A^T B (B^T B)^{-1} B^T A)^{-1} A^T - (A^T A - A^T B (B^T B)^{-1} B^T A)^{-1} A^T B (B^T B)^{-1} B^T) y $$ Now note that $$ A^T B(B^T B)^{-1} B^T = A^T Q Q^T $$ Substituting, we have $$ \hat{\alpha} = (A^T y - A^T Q Q^T y) (A^T - A^T Q Q^T A)^{-1} $$ as desired.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.