All Questions
661 questions
134
votes
14
answers
81k
views
Maximum Likelihood Estimation (MLE) in layman terms
Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
- 1,839
32
votes
4
answers
6k
views
Maximum likelihood function for mixed type distribution
In general we maximize a function
$$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$
where $f$ is probability density function if the underlying distribution is continuous, and a ...
- 1,115
21
votes
2
answers
11k
views
Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?
Background:
I'm trying to follow Princeton's review of MLE estimation for GLM.
I understand the basics of MLE estimation: likelihood, ...
- 3,360
5
votes
1
answer
1k
views
The most general definition of the Likelihood function for continuous data (including truncation and censoring)
How would you rigorously define the likelihood function for censored/truncated observations? Even in most lifetime/reliability literature (where these types of observations are frequently encountered) ...
- 691
20
votes
2
answers
6k
views
How to construct a cross-entropy loss for general regression targets?
It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as simply "cross-entropy." However, this terminology is ambiguous because different probability ...
- 94.1k
97
votes
3
answers
113k
views
What is "restricted maximum likelihood" and when should it be used?
I have read in the abstract of this paper that:
"The maximum likelihood (ML) procedure of Hartley aud Rao is modified by adapting a transformation from Patterson and Thompson which partitions the ...
- 3,942
92
votes
2
answers
104k
views
Basic question about Fisher Information matrix and relationship to Hessian and standard errors
Ok, this is a quite basic question, but I am a little bit confused. In my thesis I write:
The standard errors can be found by calculating the inverse of the square root of the diagonal elements of ...
- 1,600
32
votes
3
answers
17k
views
Extreme Value Theory - Show: Normal to Gumbel
The Maximum of $X_1,\dots,X_n. \sim$ i.i.d. Standardnormals converges to the Standard Gumbel Distribution according to Extreme Value Theory.
How can we show that?
We have
$$P(\max X_i \leq x) = P(...
- 1,271
16
votes
2
answers
3k
views
Estimating parameters of a binomial model
First of all I'd like to precise that I'm not an expert of the subject.
Suppose to have two random variables $X$ and $Y$ that are binomial, respectively $X\sim B(n_1,p)$ and $Y\sim B(n_2,p),$ note ...
- 261
59
votes
3
answers
13k
views
What kind of information is Fisher information?
Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
- 4,453
18
votes
5
answers
4k
views
Can the empirical Hessian of an M-estimator be indefinite?
Jeffrey Wooldridge in his Econometric Analysis of Cross Section and Panel Data (page 357) says that the empirical Hessian "is not guaranteed to be positive definite, or even positive semidefinite, for ...
22
votes
3
answers
4k
views
Linear regression: any non-normal distribution giving identity of OLS and MLE?
This question is inspired from the long discussion in comments here: How does linear regression use the normal distribution?
In the usual linear regression model, for simplicity here written with ...
- 82.8k
20
votes
1
answer
11k
views
the relationship between maximizing the likelihood and minimizing the cross-entropy
There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?
- 5,282
18
votes
1
answer
13k
views
ML estimate of exponential distribution (with censored data)
In Survival Analysis, you assume the survival time of a r.v. $X_i$ to be exponentially distributed. Considering now that I have $x_1,\dots,x_n$ "outcomes" of i.i.d r.v.'s $X_i$. Only some proportion ...
- 691
73
votes
4
answers
191k
views
How do you calculate the probability density function of the maximum of a sample of IID uniform random variables?
Given the random variable
$$Y = \max(X_1, X_2, \ldots, X_n)$$
where $X_i$ are IID uniform variables, how do I calculate the PDF of $Y$?
- 871
71
votes
9
answers
32k
views
Advanced statistics books recommendation
There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
10
votes
1
answer
3k
views
MLE/Likelihood of lognormally distributed interval
I have a variable set of responses that are expressed as an interval such as the sample below.
...
- 103
43
votes
1
answer
19k
views
Maximum likelihood estimators for a truncated distribution
Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
- 707
35
votes
2
answers
54k
views
REML or ML to compare two mixed effects models with differing fixed effects, but with the same random effect?
Background: Note: My data set and R code are included below text
I wish to use AIC to compare two mixed effects models generated using the lme4 package in R. Each ...
- 535
30
votes
2
answers
11k
views
Can we use MLE to estimate Neural Network weights?
I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural ...
- 403
25
votes
3
answers
9k
views
Idea and intuition behind quasi maximum likelihood estimation (QMLE)
Question(s): What is the idea and intuition behind quasi maximum likelihood estimation (QMLE; also known as pseudo maximum likelihood estimation, PMLE)? What makes the estimator work when the actual ...
- 69.5k
18
votes
1
answer
5k
views
Properties of logistic regressions
We're working with some logistic regressions and we have realized that the average estimated probability always equals the proportion of ones in the sample; that is, the average of fitted values ...
- 181
12
votes
5
answers
7k
views
What makes mean square error so good? [duplicate]
Our statistical inference course material states the following:
The principle of mean square error can be derived from the principle
of maximum likelihood (after we set a linear model where ...
- 1,309
3
votes
1
answer
622
views
How to find quantiles and likelihoods of mixture distributions?
My PDF:
M was estimated and found to be 5.
I need to work out the quartiles for the PDF above. In addition, I need to use different methods of estimation to estimate the parameters. So far I've ...
83
votes
3
answers
105k
views
How is the minimum of a set of IID random variables distributed?
If $X_1, ..., X_n$ are independent identically-distributed random variables, what can be said about the distribution of $\min(X_1, ..., X_n)$ in general?
- 941
17
votes
1
answer
10k
views
Bias of maximum likelihood estimators for logistic regression
I would like to understand a couple of fact on maximum likelihood estimators (MLEs) for logistic regressions.
Is it true that, in general, the MLE for logistic regression is biased? I would say "yes"....
- 680
11
votes
2
answers
17k
views
How does a uniform prior lead to the same estimates from maximum likelihood and mode of posterior?
I am studying different point estimate methods and read that when using MAP vs ML estimates, when we use a "uniform prior", the estimates are identical. Can somebody explain what a "uniform" prior is ...
- 229
74
votes
5
answers
106k
views
Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?
This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'.
My question is, since maximizing log likelihood is equivalent ...
- 1,823
32
votes
4
answers
27k
views
Estimating parameters of Student's t-distribution
What are the maximum-likelihood estimators for the parameters of Student's t-distribution? Do they exist in closed form? A quick Google search didn't give me any results.
Today I am interested in the ...
- 755
24
votes
2
answers
11k
views
Distribution of the maximum of two correlated normal variables
Say I have two standard normal random variables $X_1$ and $X_2$ that are jointly
normal with correlation coefficient $r$.
What is the distribution function of $\max(X_1, X_2)$?
- 2,295
22
votes
1
answer
8k
views
Seeking a Theoretical Understanding of Firth Logistic Regression
I am trying to understand Firth logistic regression (method of handling perfect/complete or quasi-complete separation in logistic regression) so I can explain it to others in simplified terms. Does ...
- 435
18
votes
3
answers
8k
views
Why does one have to use REML (instead of ML) for choosing among nested var-covar models?
Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
- 1,704
40
votes
5
answers
191k
views
How to derive the likelihood function for binomial distribution for parameter estimation?
According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as
$L(p) = \...
- 1,092
24
votes
2
answers
6k
views
Which distributions have closed-form solutions for maximum likelihood estimation?
Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?
- 342
19
votes
1
answer
30k
views
Weibull distribution parameters $k$ and $c$ for wind speed data
Hi can the same be shown to obtain shape and scale parameter for modified maximum likelihood method
- 307
80
votes
4
answers
142k
views
Maximum likelihood method vs. least squares method
What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimaton (LSE) ?
Why can't we use MLE for predicting $y$ values in linear regression and vice versa?
Any ...
- 901
29
votes
1
answer
44k
views
In R, given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix?
Given an output from optim with a hessian matrix, how to calculate parameter confidence intervals using the hessian matrix?
...
- 1,733
11
votes
1
answer
4k
views
Using bootstrap to obtain sampling distribution of 1st-percentile
I have a sample (of size 250) from a population. I do not know the distribution of the population.
The main question: I want a point estimate of the 1st-percentile of the population, and then I want ...
- 69.5k
7
votes
1
answer
3k
views
Likelihood comparable across different distributions
Suppose we have a linear model for a dependent variable $y$ in terms of two independent variables $x_1$ and $x_2$, given by $y_i=x_{i1} \beta_1+x_{i2}\beta_2+\epsilon_i$.
If we were to estimate the ...
- 829
7
votes
1
answer
10k
views
Invariance property of maximum likelihood estimator?
Here is an excerpt from one of the stats books I have been reading:
But as a counter example, let's suppose we have five possible values for $\theta$ and $\theta_5$ is the ML estimate, with the ...
- 2,828
6
votes
3
answers
1k
views
Computing the Variance of an MLE
Suppose we have i.i.d. $n$ observations $(X_1,X_2,...X_n)$ from a
population with density $$f_\theta(x)=\begin{cases}\theta x^{\theta-1}&\text{ if }0\leq x\leq 1\\0&\text{otherwise.}\end{...
- 1,825
71
votes
2
answers
47k
views
What does the inverse of covariance matrix say about data? (Intuitively)
I'm curious about the nature of $\Sigma^{-1}$. Can anybody tell something intuitive about "What does $\Sigma^{-1}$ say about data?"
Edit:
Thanks for replies
After taking some great courses, I'd ...
- 973
61
votes
8
answers
12k
views
Examples where method of moments can beat maximum likelihood in small samples?
Maximum likelihood estimators (MLE) are asymptotically efficient; we see the practical upshot in that they often do better than method of moments (MoM) estimates (when they differ), even at small ...
- 290k
22
votes
2
answers
12k
views
Why exactly is the observed Fisher information used?
In the standard maximum likelihood setting (iid sample $Y_{1}, \ldots, Y_{n}$ from some distribution with density $f_{y}(y|\theta_{0}$)) and in case of a correctly specified model the Fisher ...
- 593
22
votes
6
answers
35k
views
Fitting t-distribution in R: scaling parameter
How do I fit the parameters of a t-distribution, i.e. the parameters corresponding to the 'mean' and 'standard deviation' of a normal distribution. I assume they are called 'mean' and 'scaling/degrees ...
- 1,149
19
votes
1
answer
16k
views
What are the regularity conditions for Likelihood Ratio test
Could anyone please tell me what the regularity conditions are for the asymptotic distribution of Likelihood Ratio test?
Everywhere I look, it is written 'Under the regularity conditions' or 'under ...
- 373
16
votes
2
answers
8k
views
Generalized log likelihood ratio test for non-nested models
I understand that if I have two models A and B and A is nested in B then, given some data, I can fit the parameters of A and B using MLE and apply the generalized log likelihood ratio test. In ...
- 2,077
11
votes
4
answers
8k
views
Is Maximum Likelihood Estimation (MLE) a parametric approach?
There are two main probabilistic approaches to novelty detection: parametric and non-parametric. The non-parametric approach assumes that the distribution or density function is derived from the ...
- 113
8
votes
2
answers
922
views
In MLE for continuous rv, why is it ok to evaluate a pdf at a point?
In MLE for continuous case, my course notes define the likelihood function to be:
$$
L(\theta) = L(\theta;y) = \prod_{i=1}^n f(y_i;\theta)
$$
Where $f$ is the joint pdf of $y_i$ given $\theta$.
I ...
- 733
5
votes
2
answers
7k
views
Is least squares the standard method to fit a 3 parameters Gaussian function to some x and y data?
A participant in one experiment needs to decide whether a flash and a sound are simultaneous or not for many possible asynchronies between the flash and the sound (x in seconds). For each asynchrony, ...
- 391