Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

Filter by
Sorted by
Tagged with
117
votes
14answers
69k views

Maximum Likelihood Estimation (MLE) in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
86
votes
3answers
87k views

What is "restricted maximum likelihood" and when should it be used?

I have read in the abstract of this paper that: "The maximum likelihood (ML) procedure of Hartley aud Rao is modified by adapting a transformation from Patterson and Thompson which partitions the ...
77
votes
2answers
80k views

Basic question about Fisher Information matrix and relationship to Hessian and standard errors

Ok, this is a quite basic question, but I am a little bit confused. In my thesis I write: The standard errors can be found by calculating the inverse of the square root of the diagonal elements of ...
66
votes
5answers
84k views

Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
66
votes
3answers
97k views

Maximum likelihood method vs. least squares method

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ? Why can't we use MLE for predicting $y$ values in linear regression and vice versa? Any ...
65
votes
2answers
27k views

What is the difference between a partial likelihood, profile likelihood and marginal likelihood?

I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
63
votes
3answers
93k views

What is the difference in Bayesian estimate and maximum likelihood estimate?

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?
63
votes
9answers
25k views

Advanced statistics books recommendation

There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
62
votes
2answers
34k views

What does the inverse of covariance matrix say about data? (Intuitively)

I'm curious about the nature of $\Sigma^{-1}$. Can anybody tell something intuitive about "What does $\Sigma^{-1}$ say about data?" Edit: Thanks for replies After taking some great courses, I'd ...
58
votes
8answers
10k views

Examples where method of moments can beat maximum likelihood in small samples?

Maximum likelihood estimators (MLE) are asymptotically efficient; we see the practical upshot in that they often do better than method of moments (MoM) estimates (when they differ), even at small ...
54
votes
2answers
74k views

Cross-Entropy or Log Likelihood in Output layer

I read this page: http://neuralnetworksanddeeplearning.com/chap3.html and it said that sigmoid output layer with cross-entropy is quite similiar with softmax output layer with log-likelihood. what ...
53
votes
2answers
5k views

Intuition behind why Stein's paradox only applies in dimensions $\ge 3$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
53
votes
9answers
3k views

Are all models useless? Is any exact model possible -- or useful?

This question has been festering in my mind for over a month. The February 2015 issue of Amstat News contains an article by Berkeley Professor Mark van der Laan that scolds people for using inexact ...
41
votes
3answers
8k views

What kind of information is Fisher information?

Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
38
votes
1answer
3k views

Why does glmer not achieve the maximum likelihood (as verified by applying further generic optimization)?

Numerically deriving the MLEs of GLMM is difficult and, in practice, I know, we should not use brute force optimization (e.g., using optim in a simple way). But for ...
36
votes
3answers
55k views

Maximum Likelihood Estimators - Multivariate Gaussian

Context The Multivariate Gaussian appears frequently in Machine Learning and the following results are used in many ML books and courses without the derivations. Given data in form of a matrix $\...
34
votes
1answer
15k views

Maximum likelihood estimators for a truncated distribution

Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
31
votes
4answers
131k views

How to derive the likelihood function for binomial distribution for parameter estimation?

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as $L(p) = \...
30
votes
2answers
21k views

When should I *not* use R's nlm function for MLE?

I've run across a couple guides suggesting that I use R's nlm for maximum likelihood estimation. But none of them (including R's documentation) gives much theoretical guidance for when to use or not ...
29
votes
2answers
20k views

What is the difference between Maximum Likelihood Estimation & Gradient Descent?

What are the pro & cons of both the methods?
29
votes
4answers
6k views

Intuitive reasoning behind biased maximum likelihood estimators

I have a confusion on biased maximum likelihood (ML) estimators. The mathematics of the whole concept is pretty clear to me but I cannot figure out the intuitive reasoning behind it. Given a certain ...
28
votes
6answers
10k views

Likelihood - Why multiply?

I am studying about maximum likelihood estimation and I read that the likelihood function is the product of the probabilities of each variable. Why is it the product? Why not the sum? I have been ...
28
votes
2answers
41k views

REML or ML to compare two mixed effects models with differing fixed effects, but with the same random effect?

Background: Note: My data set and R code are included below text I wish to use AIC to compare two mixed effects models generated using the lme4 package in R. Each ...
28
votes
4answers
20k views

Estimating parameters of Student's t-distribution

What are the maximum-likelihood estimators for the parameters of Student's t-distribution? Do they exist in closed form? A quick Google search didn't give me any results. Today I am interested in the ...
28
votes
3answers
3k views

What are some illustrative applications of empirical likelihood?

I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012). In my efforts to understand it, I have gleaned ...
28
votes
4answers
17k views

Is unbiased maximum likelihood estimator always the best unbiased estimator?

I know for regular problems, if we have a best regular unbiased estimator, it must be the maximum likelihood estimator (MLE). But generally, if we have an unbiased MLE, would it also be the best ...
27
votes
2answers
24k views

Full information maximum likelihood for missing data in R

Context: Hierarchical regression with some missing data. Question: How do I use full information maximum likelihood (FIML) estimation to address missing data in R? Is there a package you would ...
26
votes
5answers
6k views

Maximum Likelihood Estimation -- why it is used despite being biased in many cases

Maximum likelihood estimation often results into biased estimators (e.g., its estimate for the sample variance is biased for the Gaussian distribution). What then makes it so popular? Why exactly is ...
26
votes
1answer
37k views

In R, given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix?

Given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix? ...
26
votes
4answers
7k views

Why maximum likelihood and not expected likelihood?

Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
26
votes
4answers
6k views

How to ensure properties of covariance matrix when fitting multivariate normal model using maximum likelihood?

Suppose I have the following model $$y_i=f(x_i,\theta)+\varepsilon_i$$ where $y_i\in \mathbb{R}^K$ , $x_i$ is a vector of explanatory variables, $\theta$ is the parameters of non-linear function $...
25
votes
2answers
9k views

Can we use MLE to estimate Neural Network weights?

I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural ...
25
votes
2answers
1k views

Fitting custom distributions by MLE

My question relates to fitting custom distributions in R but I feel it has enough of a probability element to remain on CV. I have an interesting set of data which has the following characteristics: ...
24
votes
4answers
11k views

Is there always a maximizer for any MLE problem?

I wonder if there is always a maximizer for any maximum (log-)likelihood estimation problem? In other words, is there some distribution and some of its parameters, for which the MLE problem does not ...
23
votes
2answers
21k views

How to understand that MLE of Variance is biased in a Gaussian distribution?

I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased? formula 1.55: $$ \mu_{...
21
votes
2answers
30k views

What is meant by the standard error of a maximum likelihood estimate?

I'm a mathematician self-studying statistics and struggling especially with the language. In the book I'm using, there is the following problem: A random variable $X$ is given as $\text{Pareto}(\...
21
votes
2answers
5k views

Which distributions have closed-form solutions for maximum likelihood estimation?

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?
21
votes
2answers
5k views

Why is maximum likelihood estimation considered to be a frequentist technique

Frequentist statistics for me is synonymous for trying to make decision that are good for all possible samples. I.e., a frequentist decision rule $\delta$ should always try to minimize the frequentist ...
21
votes
2answers
3k views

What are the disadvantages of the profile likelihood?

Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter. If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed ...
20
votes
6answers
18k views

Is there an example where MLE produces a biased estimate of the mean?

Can you provide an example of an MLE estimator of the mean that is biased? I am not looking for an example that breaks MLE estimators in general by violating regularity conditions. All examples I ...
20
votes
6answers
16k views

What is the 'fundamental' idea of machine learning for estimating parameters?

The 'fundamental' idea of statistics for estimating parameters is maximum likelihood. I am wondering what is the corresponding idea in machine learning. Qn 1. Would it be fair to say that the '...
20
votes
4answers
2k views

Maximum likelihood function for mixed type distribution

In general we maximize a function $$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$ where $f$ is probability density function if the underlying distribution is continuous, and a ...
19
votes
5answers
18k views

Using lmer for prediction

Hello I have two problems that sound like natural candidates for multilevel/mixed models, which I have never used. The simpler, and one that I hope to try as an introduction, is as follows: The data ...
19
votes
3answers
15k views

Does MLE require i.i.d. data? Or just independent parameters?

Estimating parameters using maximum likelihood estimation (MLE) involves evaluating the likelihood function, which maps the probability of the sample (X) occurring to values (x) on the parameter space ...
19
votes
1answer
28k views

Weibull distribution parameters $k$ and $c$ for wind speed data

Hi can the same be shown to obtain shape and scale parameter for modified maximum likelihood method
19
votes
3answers
1k views

How to do estimation, when only summary statistics are available?

This is in part motivated by the following question and the discussion following it. Suppose the iid sample is observed, $X_i\sim F(x,\theta)$. The goal is to estimate $\theta$. But original sample ...
19
votes
1answer
7k views

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Background: I'm trying to follow Princeton's review of MLE estimation for GLM. I understand the basics of MLE estimation: likelihood, ...
19
votes
2answers
8k views

Why exactly is the observed Fisher information used?

In the standard maximum likelihood setting (iid sample $Y_{1}, \ldots, Y_{n}$ from some distribution with density $f_{y}(y|\theta_{0}$)) and in case of a correctly specified model the Fisher ...
19
votes
4answers
5k views

Finding the MLE for a univariate exponential Hawkes process

The univariate exponential Hawkes process is a self-exciting point process with an event arrival rate of: $ \lambda(t) = \mu + \sum\limits_{t_i<t}{\alpha e^{-\beta(t-t_i)}}$ where $ t_1,..t_n $ ...
18
votes
5answers
2k views

What regression/estimation is not a MLE?

I just rigorously learned that OLS is a special case of MLE. It surprises me because the popular and "reliable" sources such as researchgate and this do not mention this most important connection ...

1
2 3 4 5
59