Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

Filter by
Sorted by
Tagged with
117
votes
14answers
69k views

Maximum Likelihood Estimation (MLE) in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
19
votes
1answer
7k views

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Background: I'm trying to follow Princeton's review of MLE estimation for GLM. I understand the basics of MLE estimation: likelihood, ...
20
votes
4answers
2k views

Maximum likelihood function for mixed type distribution

In general we maximize a function $$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$ where $f$ is probability density function if the underlying distribution is continuous, and a ...
41
votes
3answers
8k views

What kind of information is Fisher information?

Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
77
votes
2answers
80k views

Basic question about Fisher Information matrix and relationship to Hessian and standard errors

Ok, this is a quite basic question, but I am a little bit confused. In my thesis I write: The standard errors can be found by calculating the inverse of the square root of the diagonal elements of ...
86
votes
3answers
87k views

What is "restricted maximum likelihood" and when should it be used?

I have read in the abstract of this paper that: "The maximum likelihood (ML) procedure of Hartley aud Rao is modified by adapting a transformation from Patterson and Thompson which partitions the ...
17
votes
5answers
3k views

Can the empirical Hessian of an M-estimator be indefinite?

Jeffrey Wooldridge in his Econometric Analysis of Cross Section and Panel Data (page 357) says that the empirical Hessian "is not guaranteed to be positive definite, or even positive semidefinite, for ...
15
votes
2answers
3k views

How to construct a cross-entropy loss for general regression targets?

It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as simply "cross-entropy." However, this terminology is ambiguous because different probability ...
63
votes
9answers
25k views

Advanced statistics books recommendation

There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
12
votes
1answer
2k views

Estimating parameters for a binomial

First of all I'd like to precise that I'm not an expert of the subject. Suppose to have two random variables $X$ and $Y$ that are binomial, respectively $X\sim B(n_1,p)$ and $Y\sim B(n_2,p),$ note ...
12
votes
1answer
9k views

ML estimate of exponential distribution (with censored data)

In Survival Analysis, you assume the survival time of a r.v. $X_i$ to be exponentially distributed. Considering now that I have $x_1,\dots,x_n$ "outcomes" of i.i.d r.v.'s $X_i$. Only some proportion ...
13
votes
1answer
5k views

the relationship between maximizing the likelihood and minimizing the cross-entropy

There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?
17
votes
1answer
4k views

Properties of logistic regressions

We're working with some logistic regressions and we have realized that the average estimated probability always equals the proportion of ones in the sample; that is, the average of fitted values ...
34
votes
1answer
15k views

Maximum likelihood estimators for a truncated distribution

Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
18
votes
3answers
6k views

Idea and intuition behind quasi maximum likelihood estimation (QMLE)

Question(s): What is the idea and intuition behind quasi maximum likelihood estimation (QMLE; also known as pseudo maximum likelihood estimation, PMLE)? What makes the estimator work when the actual ...
10
votes
2answers
12k views

How does a uniform prior lead to the same estimates from maximum likelihood and mode of posterior?

I am studying different point estimate methods and read that when using MAP vs ML estimates, when we use a "uniform prior", the estimates are identical. Can somebody explain what a "uniform" prior is ...
17
votes
3answers
5k views

Why does one have to use REML (instead of ML) for choosing among nested var-covar models?

Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
19
votes
1answer
28k views

Weibull distribution parameters $k$ and $c$ for wind speed data

Hi can the same be shown to obtain shape and scale parameter for modified maximum likelihood method
14
votes
1answer
7k views

Bias of maximum likelihood estimators for logistic regression

I would like to understand a couple of fact on maximum likelihood estimators (MLEs) for logistic regressions. Is it true that, in general, the MLE for logistic regression is biased? I would say "yes"....
7
votes
1answer
8k views

Invariance property of maximum likelihood estimator?

Here is an excerpt from one of the stats books I have been reading: But as a counter example, let's suppose we have five possible values for $\theta$ and $\theta_5$ is the ML estimate, with the ...
28
votes
2answers
41k views

REML or ML to compare two mixed effects models with differing fixed effects, but with the same random effect?

Background: Note: My data set and R code are included below text I wish to use AIC to compare two mixed effects models generated using the lme4 package in R. Each ...
25
votes
2answers
9k views

Can we use MLE to estimate Neural Network weights?

I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural ...
5
votes
2answers
5k views

Is least squares the standard method to fit a 3 parameters Gaussian function to some x and y data?

A participant in one experiment needs to decide whether a flash and a sound are simultaneous or not for many possible asynchronies between the flash and the sound (x in seconds). For each asynchrony, ...
62
votes
2answers
34k views

What does the inverse of covariance matrix say about data? (Intuitively)

I'm curious about the nature of $\Sigma^{-1}$. Can anybody tell something intuitive about "What does $\Sigma^{-1}$ say about data?" Edit: Thanks for replies After taking some great courses, I'd ...
26
votes
1answer
37k views

In R, given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix?

Given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix? ...
28
votes
4answers
20k views

Estimating parameters of Student's t-distribution

What are the maximum-likelihood estimators for the parameters of Student's t-distribution? Do they exist in closed form? A quick Google search didn't give me any results. Today I am interested in the ...
17
votes
3answers
3k views

Linear regression: any non-normal distribution giving identity of OLS and MLE?

This question is inspired from the long discussion in comments here: How does linear regression use the normal distribution? In the usual linear regression model, for simplicity here written with ...
21
votes
2answers
5k views

Which distributions have closed-form solutions for maximum likelihood estimation?

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?
8
votes
4answers
6k views

Is Maximum Likelihood Estimation (MLE) a parametric approach?

There are two main probabilistic approaches to novelty detection: parametric and non-parametric. The non-parametric approach assumes that the distribution or density function is derived from the ...
63
votes
3answers
93k views

What is the difference in Bayesian estimate and maximum likelihood estimate?

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?
13
votes
2answers
8k views

MLE of the location parameter in a Cauchy distribution

After centering, the two measurements x and −x can be assumed to be independent observations from a Cauchy distribution with probability density function: $f(x :\theta) = $ $1\over\pi (1+(x-\...
18
votes
1answer
13k views

What are the regularity conditions for Likelihood Ratio test

Could anyone please tell me what the regularity conditions are for the asymptotic distribution of Likelihood Ratio test? Everywhere I look, it is written 'Under the regularity conditions' or 'under ...
13
votes
2answers
6k views

Integrating kernel density estimator in 2D

I'm coming from this question in case anybody wants to follow the trail. Basically I have a data set $\Omega$ composed of $N$ objects where each object has a given number of measured values attached ...
12
votes
5answers
6k views

What makes mean square error so good? [duplicate]

Our statistical inference course material states the following: The principle of mean square error can be derived from the principle of maximum likelihood (after we set a linear model where ...
4
votes
2answers
4k views

What to do when your likelihood function has a double product with small values near zero - log transform doesn't work?

I currently have a likelihood function defined as the following: $$ L=\prod_{i=1}^{N}\left[\prod_{s=1}^{S_i}L_{is}(y\space|\space \rho_A)\times\phi + \prod_{s=1}^{S_i}L_{is}(y\space|\space \rho_B)\...
8
votes
1answer
2k views

MLE/Likelihood of lognormally distributed interval

I have a variable set of responses that are expressed as an interval such as the sample below. ...
66
votes
3answers
97k views

Maximum likelihood method vs. least squares method

What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ? Why can't we use MLE for predicting $y$ values in linear regression and vice versa? Any ...
66
votes
5answers
84k views

Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
58
votes
8answers
10k views

Examples where method of moments can beat maximum likelihood in small samples?

Maximum likelihood estimators (MLE) are asymptotically efficient; we see the practical upshot in that they often do better than method of moments (MoM) estimates (when they differ), even at small ...
53
votes
2answers
5k views

Intuition behind why Stein's paradox only applies in dimensions $\ge 3$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
31
votes
4answers
131k views

How to derive the likelihood function for binomial distribution for parameter estimation?

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as $L(p) = \...
21
votes
2answers
5k views

Why is maximum likelihood estimation considered to be a frequentist technique

Frequentist statistics for me is synonymous for trying to make decision that are good for all possible samples. I.e., a frequentist decision rule $\delta$ should always try to minimize the frequentist ...
19
votes
2answers
8k views

Why exactly is the observed Fisher information used?

In the standard maximum likelihood setting (iid sample $Y_{1}, \ldots, Y_{n}$ from some distribution with density $f_{y}(y|\theta_{0}$)) and in case of a correctly specified model the Fisher ...
15
votes
1answer
5k views

Why should one use EM vs. say, Gradient Descent with MLE?

Mathematically, it's often seen that expressions and algorithms for Expectation Maximization (EM) are often simpler for mixed models, yet it seems that almost everything (if not everything) that can ...
26
votes
5answers
6k views

Maximum Likelihood Estimation -- why it is used despite being biased in many cases

Maximum likelihood estimation often results into biased estimators (e.g., its estimate for the sample variance is biased for the Gaussian distribution). What then makes it so popular? Why exactly is ...
6
votes
1answer
10k views

Maximum likelihood in the GJR-GARCH(1,1) model

In the standard GARCH(1,1) model with normal innovations $\sigma^2_t=\omega+\alpha\epsilon^2_{t-1}+\beta\sigma^2_{t-1} $ the likelihood of $m$ observations occurring in the order in which they are ...
10
votes
1answer
2k views

General theorems for consistency and asymptotic normality of maximum likelihood

I'm interested in a good reference for results concerning asymptotic properties of maximum likelihood estimators. Consider a model $\{f_n(\cdot \mid \theta): \theta \in \Theta, n \in \mathbb N\}$ ...
16
votes
1answer
7k views

Seeking a Theoretical Understanding of Firth Logistic Regression

I am trying to understand Firth logistic regression (method of handling perfect/complete or quasi-complete separation in logistic regression) so I can explain it to others in simplified terms. Does ...
18
votes
4answers
28k views

Fitting t-distribution in R: scaling parameter

How do I fit the parameters of a t-distribution, i.e. the parameters corresponding to the 'mean' and 'standard deviation' of a normal distribution. I assume they are called 'mean' and 'scaling/degrees ...
5
votes
1answer
3k views

Confidence regions on bivariate normal distributions using $\hat{\Sigma}_{MLE}$ or $\mathbf{S}$

Given a $5 \times 2$ dataset $\mathbf{X} =\left( \begin{array}{rr}-0.9&0.2\\2.4&0.7\\-1.4&1.0\\2.9&-0.5\\2.0&-1.0 \end{array} \right)$. Assume that $X\sim N_2(\mu, \Sigma)$. ...

1
2 3 4 5
9