4
$\begingroup$

In a machine learning class, the instructor was explaining Maximum Likelihood Estimation. He mentioned that

$\hat{\theta}_{MLE}=argmax_\theta P(D;\theta)$

Where D is the data observed, $\theta$ is the true estimation and $\hat{\theta}$ is the estimator.

Is an estimator the maximum value of all values that a true estimation?

Is a true estimation the same as a parameter? (this is not a second question but more of a segue from the first, which i thought should be asked here instead of in a separate question)

$\endgroup$
2
$\begingroup$

'true estimation' is, I believe, not really a common term.

What you have there $P(D;\theta)$ is the likelihood function which is also the same as the probability (density) of the data $D$ given the parameter(s) $\theta$.

So you could see the $\theta$, when used in this place, as a hypothetically true parameter. You express "Suppose the parameter is truely $\theta$ then what will be the probability (density) to observe the data $D$".

This is a trick to say something about inverse probability (see also fiducial inference and Bayes theorem). What is inverse? We can easily describe mathematically the probability to observe some data given some distribution parameters. But, the other direction does not work well, we can not easily (if possible at all) describe mathematically the probability that some distribution parameters are 'true' given the observation of some sampled data.

$\endgroup$
2
$\begingroup$

By "true estimation" I suspect you mean "true parameter".

Nevertheless, it doesn't make sense to talk about $\theta$ as the "true parameter" in the context of what you've written. The likelihood function $P(D; \theta)$ is simply a function of $\theta$, which is an optimization variable in this context.

The maximum likelihood estimator(s) is(are) the value(s) of $\theta$ which maximize the function $P(D; \theta)$. If your likelihood is strictly concave and the parameter space (i.e., the feasible set for your optimization variable $\theta$) is convex, then there is only one such value of $\theta$ which maximizes the likelihood, which is often referred to as the maximum likelihood estimator.

To relate the maximum likelihood estimator $\hat{\theta}_{MLE}$ to the "true parameter", you can think of the "true parameter" as the value of $\theta$ which would maximize the likelihood if you collected an infinite number of samples (assuming your model is correctly specified and some standard regularity conditions are met). That is, the maximum likelihood estimator is equal to the "true parameter'' as you observe data on a very, very large number of subjects (again, under some standard regularity conditions). Formalizing this notion is the basis of maximum likelihood theory.

$\endgroup$
0
$\begingroup$

I think the instructor may have meant that $\theta$ is the 'true parameter' and that $\hat{\theta}$ is the maximum-likelihood estimate for $\theta$. In other words, among all possible values that $\theta$ can potentially take $\hat{\theta}$ is the one that maximizes the likelihood of observing the data we have (viz $D$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.