1
$\begingroup$

If we use Maximum Likelihood Estimation to estimate regression parameters (B and sigma), and if we have less observations (n) than predictors (p), can we bypass dimension reduction ? My understanding is if we do least squares estimation, we can't solve for the Bs. Sorry my math is not good otherwise I would just prove it myself.

$\endgroup$
3
  • $\begingroup$ Maximum Likelihood Estimation in linear models is equivalent to least squares regression in that both amount to minimising $\sum (\hat {y}_i - y_i)^2$. In both approaches, with $n < p$ there will be multiple possible coefficients giving the same optimal results and so no unique solution. In many cases this will also be associated with serious overfitting with all the residuals $0$ $\endgroup$ – Henry May 22 '20 at 11:00
  • 1
    $\begingroup$ You can solve this problem with regularization. $\endgroup$ – Sextus Empiricus May 22 '20 at 13:35
  • 1
    $\begingroup$ I'd say the short answer is "no". $\endgroup$ – Ben Bolker May 22 '20 at 23:01
1
$\begingroup$

The answer is 'no'.

As mentioned in the comments, MLE is equivalent to least squares estimation (in the case of a Gaussian distributed noise/error), and it won't solve the problem of least squares estimation.

The problem occurs in the likelihood function being parameterized by a term for the conditional mean $\mu = \beta X$. This term $\beta X$ is an over determined system of equations, and for the same $\mu$ there are multiple solutions $\beta$. So it occurs in the likelihood function just as well as in the least squares estimation.

However your idea/thought was not so bad

It is not generally true that you can't find a maximum likelihood estimate when there are more parameters than observations. So it makes sense that you were thinking of it.

The number of observations does not really matter as the MLE problem remains a single function/equation. We can optimize single functions with many parameters, there is not in general a problem there. The problem is just in the term $\beta X$.

E.g. the likelihood related to a Binomial distribution with two parameters $n$ and $p$ can be maximized for a single observation. While this is normally a difficult problem, for a single observation the likelihood is simply maximized when $p=1$ and $n=X$. (this is an example, I am not saying that these estimates where 'n_observations < parameters' are good estimates)

Regularization

In a similar way we can solve the problem with the over-determined $\beta X$ when, $n < p$ by using regularization. So indeed there are way to solve a minimization/maximization problem when there are more parameters than observations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.