Matrix differentiation for Maximum Likelihood Estimator

Question

In trying to derive the maximum likelihood estimator for the mean, $\mu$, of a p-variate normal distribution it is needed to differentiate $\sum_{i=1}^n(x_i-\mu)^T\Sigma^{-1}(x_i-\mu)$ w.r.t the vector $\mu$. After doing this I am left with the derivative $-2\sum_{i=1}^n(x_i-\mu)^T\Sigma^{-1}$. In order to solve for $\mu$ it is necessary to equate this to zero, however I will then not solve for $\mu$ but for $\mu^T$.

Also, my lecturer seems to say that the derivative of the aforementioned expression w.r.t $\mu$ is actually the transpose of my answer, which will lead to solving for $\mu$ directly.

Somewhere I have read that the derivative can be defined in terms of either a column or row vector. Is this why my answer is different than my lecturer's answer and is it still correct to solve for $\mu^T$?

Answer 1

See the stats.stackexchange.com question linked to below. This is largely a matter of convention.

In statistics and most areas of applied mathematics, the gradient of scalar valued function of a column vector is itself a column vector, and this vector is explicitly transposed in expressions like the Taylor series. In this convention, the gradient of a function may be called "the derivative" of a function. That's that convention that your instructor is apparently using, and it would be practical to stick with this convention to keep your instructor happy.

In differential geometry and mathematical physics, many authors adopt the convention that "the derivative of f" is the the row vector of partial derivatives of f. Authors who follow this convention typically still refer to the transpose of the derivative as the gradient, so that the gradient is still a column vector. Although it may seem like this is merely a convention, the distinction actually does become useful as you start to do tensor analysis.

If this bothers you, then simply write that "We find the MLE by computing the gradient of the likelihood and setting the gradient equal to 0. This results in the equation $\sum_{i} \Sigma^{-1} (x_{i}-\mu)=0$." The quoted sentence is correct under either convention because we've explicitly referred to the gradient (column vector) rather than the derivative.

Note that it is nearly universal in statistics to use a column vector $\mu$ for the mean of a multivariate normal distribution.

Gradient and vector derivatives: row or column vector?