Skip to main content
90 votes
Accepted

Cross-Entropy or Log Likelihood in Output layer

The negative log likelihood (eq.80) is also known as the multiclass cross-entropy (ref: Pattern Recognition and Machine Learning Section 4.3.4), as they are in fact two different interpretations of ...
dontloo's user avatar
  • 16.4k
66 votes
Accepted

Maximum Likelihood Estimators - Multivariate Gaussian

Deriving the Maximum Likelihood Estimators Assume that we have $m$ random vectors, each of size $p$: $\mathbf{X^{(1)}, X^{(2)}, \dotsc, X^{(m)}}$ where each random vectors can be interpreted as an ...
Xavier Bourret Sicotte's user avatar
53 votes
Accepted

Likelihood - Why multiply?

This is a very basic question, and instead of using formal language and mathematical notation, I will try to answer it at a level at which everybody who can understand the question can also understand ...
rumtscho's user avatar
  • 1,819
44 votes

What kind of information is Fisher information?

Let's think in terms of the negative log-likelihood function $\ell$. The negative score is its gradient with respect to the parameter value. At the true parameter, the score is zero. Otherwise, it ...
Neil G's user avatar
  • 15.3k
42 votes

Maximum likelihood method vs. least squares method

I'd like to provide a straightforward answer. What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimation (LSE) ? As @TrynnaDoStat commented, minimizing ...
Lerner Zhang's user avatar
  • 6,646
41 votes
Accepted

What kind of information is Fisher information?

Trying to complement the other answers... What kind of information is Fisher information? Start with the loglikelihood function $$ \ell (\theta) = \log f(x;\theta) $$ as a function of $\theta$ for ...
kjetil b halvorsen's user avatar
40 votes
Accepted

Is there an example where MLE produces a biased estimate of the mean?

Christoph Hanck has not posted the details of his proposed example. I take it he means the uniform distribution on the interval $[0,\theta],$ based on an i.i.d. sample $X_1,\ldots,X_n$ of size more ...
Michael Hardy's user avatar
28 votes
Accepted

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Some years ago I wrote a paper about this for my students (in spanish), so I can try to rewrite those explanations here. I will look at IRLS (iteratively reweighted least squares) through a series of ...
kjetil b halvorsen's user avatar
28 votes
Accepted

What regression/estimation is not a MLE?

Least squares is indeed maximum likelihood if the errors are iid normal, but if they aren't iid normal, least squares is not maximum likelihood. For example if my errors were logistic, least squares ...
Glen_b's user avatar
  • 283k
27 votes

Why is everything based on likelihoods even though likelihoods are so small?

The key lies not in the absolute size of the likelihood values but in their relative comparison and the mathematical principles underlying likelihood-based methods. The smallness of the likelihood is ...
ADAM's user avatar
  • 731
26 votes

Cross-Entropy or Log Likelihood in Output layer

Expanding on @dontloo's answer, consider a classification task with $K$ classes. Let's separately look at the output layer of a network and the cost function. For our purpose here, the output layer ...
user650654's user avatar
26 votes
Accepted

Do we ever use maximum likelihood estimation?

I am wondering if maximum likelihood estimation ever used in statistics. Certainly! Actually quite a lot -- but not always. We learn the concept of it but I wonder when it is actually used. When ...
Glen_b's user avatar
  • 283k
26 votes

Maximum Likelihood Estimation for Bernoulli distribution

Its often easier to work with the log-likelihood in these situations than the likelihood. Note that the minimum/maximum of the log-likelihood is exactly the same as the min/max of the likelihood. $$ \...
bdeonovic's user avatar
  • 10.1k
25 votes

What kind of information is Fisher information?

Complementary to @NeilG's nice answer (+1) and to address your specific questions: I would say it counts the "precision" rather than the "error" itself. Remember that the Hessian of the log-...
usεr11852's user avatar
  • 44.2k
24 votes

MLE convergence errors with statespace SARIMAX

First, mle_retvals should be an attribute of SARIMAXResults if it is constructed using a fit ...
cfulton's user avatar
  • 1,463
23 votes

Is unbiased maximum likelihood estimator always the best unbiased estimator?

But generally, if we have an unbiased MLE, would it also be the best unbiased estimator ? If there is a complete sufficient statistics, yes. Proof: Lehmann–Scheffé theorem: Any unbiased ...
Benoit Sanchez's user avatar
23 votes
Accepted

Relation between MAP, EM, and MLE

Imagine that you have some data $X$ and probabilistic model parametrized by $\theta$, you are interested in learning about $\theta$ given your data. The relation between data, parameter and model is ...
Tim's user avatar
  • 138k
23 votes

Maximum likelihood function for mixed type distribution

I admit to puzzling over this question for quite some time earlier in my career. One way I convinced myself of the answer was to take an extremely practical, applied view of the situation, a view ...
whuber's user avatar
  • 324k
23 votes

Why are the Least-Squares and Maximum-Likelihood methods of regression not equivalent when the errors are not normally distributed?

Short Answer The probability density of a multivariate Gaussian distributed variable $x=(x_1, x_2,...,x_n)$, with mean $\mu=(\mu_1,\mu_2,...,\mu_n)$ is related to the square of the euclidean distance ...
Sextus Empiricus's user avatar
22 votes

Likelihood - Why multiply?

Independence between two events means that the occurrence of one event does not affect the likelihood of the occurrence of the another event . So for any two events $A$ and $B$ in a sample space $S$ ...
Bahgat Nassour's user avatar
22 votes
Accepted

Relationship between Hessian Matrix and Covariance Matrix

You should first check out this: Basic question about Fisher Information matrix and relationship to Hessian and standard errors. Suppose we have a statistical model (family of distributions) $\{f_{\...
Łukasz Grad's user avatar
  • 2,198
21 votes
Accepted

simulating random samples with a given MLE

One option would be to use a constrained HMC variant as described in A Family of MCMC Methods on Implicitly Defined Manifolds by Brubaker et al (1). This requires that we can express the condition ...
Matt Graham's user avatar
21 votes

the relationship between maximizing the likelihood and minimizing the cross-entropy

Here's a worked example in the case of iid binary data, each with a success/failure recorded as $y_i \in \{0,1\}$. For labels $y_i\in \{0,1\}$, the likelihood of some binary data under the Bernoulli ...
Sycorax's user avatar
  • 91k
20 votes

Are all models useless? Is any exact model possible -- or useful?

The cited article seems to be based on fears that statisticians "will not be an intrinsic part of the scientific team, and the scientists will naturally have their doubts about the methods used" and ...
EdM's user avatar
  • 92.5k
20 votes
Accepted

When do maximum likelihood and method of moments produce the same estimators?

A general answer is that an estimator based on a method of moments is not invariant by a bijective change of parameterisation, while a maximum likelihood estimator is invariant. Therefore, they almost ...
Xi'an's user avatar
  • 106k
20 votes
Accepted

Why does MLE make sense, given the probability of an individual sample is 0?

The probability of any sample, $\mathbb{P}_\theta(X=x)$, is equal to zero and yet one sample is realised by drawing from a probability distribution. Probability is therefore the wrong tool for ...
Xi'an's user avatar
  • 106k
20 votes
Accepted

Finding category with maximum likelihood method

This is a classic unsupervised learning problem that has a simple maximum likelihood solution. The solution is a motivating example for the expectation maximization algorithm. The process is: ...
AdamO's user avatar
  • 62.7k
19 votes
Accepted

Fitting t-distribution in R: scaling parameter

fitdistr uses maximum-likelihood and optimization techniques to find parameters of a given distribution. Sometimes, especially for t-distribution, as @user12719 ...
Sergey Bushmanov's user avatar
19 votes

Maximum likelihood function for mixed type distribution

This question is an extremely important foundational problem in likelihood analysis, and also a very subtle and difficult one, so I'm quite surprised at some of the superficial answers it is receiving ...
Ben's user avatar
  • 125k
19 votes

Is there an example where MLE produces a biased estimate of the mean?

Here's an example that I think some may find surprising: In logistic regression, for any finite sample size with non-deterministic outcomes (i.e. $0 < p_{i} < 1$), any estimated regression ...
Cliff AB's user avatar
  • 21.1k

Only top scored, non community-wiki answers of a minimum length are eligible