Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

Filter by
Sorted by
Tagged with
90
votes
11answers
54k views

Maximum Likelihood Estimation (MLE) in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
73
votes
3answers
60k views

What is “restricted maximum likelihood” and when should it be used?

I have read in the abstract of this paper that: "The maximum likelihood (ML) procedure of Hartley aud Rao is modified by adapting a transformation from Patterson and Thompson which partitions the ...
57
votes
7answers
8k views

Examples where method of moments can beat maximum likelihood in small samples?

Maximum likelihood estimators (MLE) are asymptotically efficient; we see the practical upshot in that they often do better than method of moments (MoM) estimates (when they differ), even at small ...
56
votes
2answers
23k views

What is the difference between a partial likelihood, profile likelihood and marginal likelihood?

I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
53
votes
2answers
57k views

Basic question about Fisher Information matrix and relationship to Hessian and standard errors

Ok, this is a quite basic question, but I am a little bit confused. In my thesis I write: The standard errors can be found by calculating the inverse of the square root of the diagonal elements of ...
53
votes
9answers
17k views

Advanced statistics books recommendation

There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
47
votes
3answers
64k views

What is the difference in Bayesian estimate and maximum likelihood estimate?

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?
46
votes
2answers
4k views

Intuition behind why Stein's paradox only applies in dimensions $\ge 3$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
46
votes
2answers
24k views

What does the inverse of covariance matrix say about data? (Intuitively)

I'm curious about the nature of $\Sigma^{-1}$. Can anybody tell something intuitive about "What does $\Sigma^{-1}$ say about data?" Edit: Thanks for replies After taking some great courses, I'd ...
43
votes
4answers
59k views

I am wondering why we use negative (log) likelihood sometimes?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
42
votes
8answers
2k views

Are all models useless? Is any exact model possible — or useful?

This question has been festering in my mind for over a month. The February 2015 issue of Amstat News contains an article by Berkeley Professor Mark van der Laan that scolds people for using inexact ...
40
votes
2answers
50k views

Maximum likelihood method vs. least squares method

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ? Why can't we use MLE for predicting $y$ values in linear regression and vice versa? Any ...
37
votes
1answer
2k views

Why does glmer not achieve the maximum likelihood (as verified by applying further generic optimization)?

Numerically deriving the MLEs of GLMM is difficult and, in practice, I know, we should not use brute force optimization (e.g., using optim in a simple way). But for ...
28
votes
3answers
3k views

What kind of information is Fisher information?

Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
28
votes
3answers
3k views

What are some illustrative applications of empirical likelihood?

I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012). In my efforts to understand it, I have gleaned ...
27
votes
1answer
11k views

Maximum likelihood estimators for a truncated distribution

Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
26
votes
1answer
41k views

Cross-Entropy or Log Likelihood in Output layer

I read this page: http://neuralnetworksanddeeplearning.com/chap3.html and it said that sigmoid output layer with cross-entropy is quite similiar with softmax output layer with log-likelihood. what ...
25
votes
5answers
4k views

Maximum Likelihood Estimation — why it is used despite being biased in many cases

Maximum likelihood estimation often results into biased estimators (e.g., its estimate for the sample variance is biased for the Gaussian distribution). What then makes it so popular? Why exactly is ...
25
votes
3answers
3k views

Intuitive reasoning behind biased maximum likelihood estimators

I have a confusion on biased maximum likelihood (ML) estimators. The mathematics of the whole concept is pretty clear to me but I cannot figure out the intuitive reasoning behind it. Given a certain ...
24
votes
2answers
18k views

When should I *not* use R's nlm function for MLE?

I've run across a couple guides suggesting that I use R's nlm for maximum likelihood estimation. But none of them (including R's documentation) gives much theoretical guidance for when to use or not ...
23
votes
4answers
9k views

Is there always a maximizer for any MLE problem?

I wonder if there is always a maximizer for any maximum (log-)likelihood estimation problem? In other words, is there some distribution and some of its parameters, for which the MLE problem does not ...
23
votes
2answers
7k views

Can we use MLE to estimate Neural Network weights?

I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural ...
22
votes
1answer
28k views

In R, given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix?

Given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix? ...
22
votes
4answers
5k views

How to ensure properties of covariance matrix when fitting multivariate normal model using maximum likelihood?

Suppose I have the following model $$y_i=f(x_i,\theta)+\varepsilon_i$$ where $y_i\in \mathbb{R}^K$ , $x_i$ is a vector of explanatory variables, $\theta$ is the parameters of non-linear function $...
21
votes
6answers
7k views

Likelihood - Why multiply?

I am studying about maximum likelihood estimation and I read that the likelihood function is the product of the probabilities of each variable. Why is it the product? Why not the sum? I have been ...
21
votes
4answers
14k views

Estimating parameters of Student's t-distribution

What are the maximum-likelihood estimators for the parameters of Student's t-distribution? Do they exist in closed form? A quick Google search didn't give me any results. Today I am interested in the ...
21
votes
2answers
71k views

How to derive the likelihood function for binomial distribution for parameter estimation?

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as $L(p) = \...
21
votes
4answers
5k views

Is unbiased maximum likelihood estimator always the best unbiased estimator?

I know for regular problems, if we have a best regular unbiased estimator, it must be the maximum likelihood estimator (MLE). But generally, if we have an unbiased MLE, would it also be the best ...
20
votes
2answers
20k views

What is meant by the standard error of a maximum likelihood estimate?

I'm a mathematician self-studying statistics and struggling especially with the language. In the book I'm using, there is the following problem: A random variable $X$ is given as $\text{Pareto}(\...
20
votes
2answers
3k views

Which distributions have closed-form solutions for maximum likelihood estimation?

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?
19
votes
6answers
15k views

What is the 'fundamental' idea of machine learning for estimating parameters?

The 'fundamental' idea of statistics for estimating parameters is maximum likelihood. I am wondering what is the corresponding idea in machine learning. Qn 1. Would it be fair to say that the '...
19
votes
3answers
5k views

Why maximum likelihood and not expected likelihood?

Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
19
votes
1answer
24k views

Weibull distribution parameters $k$ and $c$ for wind speed data

Hi can the same be shown to obtain shape and scale parameter for modified maximum likelihood method
19
votes
1answer
2k views

What are the disadvantages of the profile likelihood?

Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter. If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed ...
18
votes
5answers
17k views

Using lmer for prediction

Hello I have two problems that sound like natural candidates for multilevel/mixed models, which I have never used. The simpler, and one that I hope to try as an introduction, is as follows: The data ...
18
votes
2answers
3k views

Why is maximum likelihood estimation considered to be a frequentist technique

Frequentist statistics for me is synonymous for trying to make decision that are good for all possible samples. I.e., a frequentist decision rule $\delta$ should always try to minimize the frequentist ...
17
votes
1answer
3k views

Properties of logistic regressions

We're working with some logistic regressions and we have realized that the average estimated probability always equals the proportion of ones in the sample; that is, the average of fitted values ...
17
votes
2answers
15k views

Maximum Likelihood Estimators - Multivariate Gaussian

Context The Multivariate Gaussian appears frequently in Machine Learning and the following results are used in many ML books and courses without the derivations. Given data in form of a matrix $\...
17
votes
1answer
457 views

simulating random samples with a given MLE

This Cross Validated question asking about simulating a sample conditional on having a fixed sum reminded me of a problem set to me by George Casella. Given a parametric model $f(x|\theta)$, and ...
17
votes
3answers
1k views

How to do estimation, when only summary statistics are available?

This is in part motivated by the following question and the discussion following it. Suppose the iid sample is observed, $X_i\sim F(x,\theta)$. The goal is to estimate $\theta$. But original sample ...
17
votes
2answers
701 views

An impossible estimation problem?

Question The variance of a negative binomial (NB) distribution is always greater than its mean. When the mean of a sample is greater than its variance, trying to fit the parameters of a NB with ...
17
votes
1answer
3k views

When do maximum likelihood and method of moments produce the same estimators?

I was asked this question the other day and had never considered it before. My intuition comes from the advantages of each estimator. Maximum likelihood is preferably when we are confident in the ...
17
votes
1answer
7k views

MLE vs least squares in fitting probability distributions

The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood ...
16
votes
6answers
9k views

Is there an example where MLE produces a biased estimate of the mean?

Can you provide an example of an MLE estimator of the mean that is biased? I am not looking for an example that breaks MLE estimators in general by violating regularity conditions. All examples I ...
16
votes
3answers
18k views

Fitting t-distribution in R: scaling parameter

How do I fit the parameters of a t-distribution, i.e. the parameters corresponding to the 'mean' and 'standard deviation' of a normal distribution. I assume they are called 'mean' and 'scaling/degrees ...
16
votes
3answers
11k views

Does MLE require i.i.d. data? Or just independent parameters?

Estimating parameters using maximum likelihood estimation (MLE) involves evaluating the likelihood function, which maps the probability of the sample (X) occurring to values (x) on the parameter space ...
16
votes
2answers
4k views

Why exactly is the observed Fisher information used?

In the standard maximum likelihood setting (iid sample $Y_{1}, \ldots, Y_{n}$ from some distribution with density $f_{y}(y|\theta_{0}$)) and in case of a correctly specified model the Fisher ...
16
votes
3answers
3k views

Idea and intuition behind quasi maximum likelihood estimation (QMLE)

Question(s): What is the idea and intuition behind quasi maximum likelihood estimation (QMLE; also known as pseudo maximum likelihood estimation, PMLE)? What makes the estimator work when the actual ...
16
votes
3answers
16k views

Full information maximum likelihood for missing data in R

Context: Hierarchical regression with some missing data. Question: How do I use full information maximum likelihood (FIML) estimation to address missing data in R? Is there a package you would ...
16
votes
3answers
5k views

Finding the MLE for a univariate exponential Hawkes process

The univariate exponential Hawkes process is a self-exciting point process with an event arrival rate of: $ \lambda(t) = \mu + \sum\limits_{t_i<t}{\alpha e^{-\beta(t-t_i)}}$ where $ t_1,..t_n $ ...