When would maximum likelihood estimates equal least squares estimates?

Question

Under what situations would MLE (Maximum Likelihood Estimates) equal LSE (Least Squares Estimates)?

I got an impression that under norm 2 ($L_{2}$), MLE and LSE are equal.

For example, the process of solving $\mathrm{min}||y−Ax||_{2}$ is actually the MLE estimation of parameter $A$ for random variable $y=Ax+\epsilon$ where $x$ and $\epsilon$ is normal.

However, is that generally true that the minimization problem under $L_{2}$ norm is the same as maximum likelihood estimation? For example, consider a quadratic function $f(X) = XAX$, can minimizing the distance between $f(X)$ and some value $Y$ under $L_{2}$ can be solved by MLE?

Answer 1

When the statistical properties of the underlying data-generating process are "normal", i.e., error terms are Gaussian distributed and iid. In this case, the maximum likelihood estimator is equivalent to the least-squares estimator.

Answer 2

In MLE there is no L2 norm. Imagine doing logistic regression (LR). This is an MLE problem. The objective function for this, and for any MLE problem, is the likelihood or the probability of Y, given X (N x p) and beta. The responses for LR are N-dimensional sequences of {H, T} or {T, F}. There is no natural way to calculate the distance between a sequence {H, H, T} and a vector of conditional means (these are what you generate from every choice of beta). Some folks will code the responses as {0, 1}, but that is artificial. The true sample space is whatever is in the problem. If it's {T, F}, then the sample space is a sequence of {T, F}'s. Add to that folks using the coded values {0, 1} as scalars in the likelihood calculation. The way I think about LR is that for every beta you generate an N-dimensional conditional mean vector. So each beta generates a pmf on the true space of responses...no coding needed. Each value is the probability of observing, to be specific, an "H". So the range of the design matrix, X, is, through the link function, a p-dimensional manifold completely contained in the open unit cube (0, 1)^N. Every point in the range of X is a pmf. Thought of in this manner, the corners of the unit cube correspond to exactly the 2^N set of possible responses. But the correspondence is that the corners are point mass pmf's that place probability 1 on the corner. They are not vectors of 0's and 1's. Finally note that the set of all pmf's on a sample space is a metric space. There are many norms available, but use total variation. What pmf in the range of X is "closest" to the point mass pmf for Y...It's the MLE. So MLE and OLS are related, but in OLS you imagine the range of X and the responses as vectors in N-dimensional euclidean space. Once you move to the glm, you use MLE to define the range of X as an N-dimensional space of pmf's with a norm (total variation). It's not a vector space since you can't add pmf's, but it is a metric space. The response, Y, points to a pmf on a corner. The pmf in the range of X that is closest to the pmf for Y tells us the MLE..just find the corresponding beta.