Questions tagged [maximum-likelihood]
a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.
3,211
questions
0
votes
1
answer
625
views
How to find the asymptotic distribution of an estimator given the mean and variance of an estimator
I understand that the Delta Method can be used to find asymptotic distribution of estimators.
I have a MLE Estimator with
$ E[\hat\Theta] = \frac{n\Theta_0}{n+1} $
$ Var[\hat\Theta] = \frac{\Theta^...
CommunityBot
- 1
2
votes
1
answer
12
views
Likelihood estimation with 2 samples from a geometric distribution
Context
Given there are 2 groups that can be modelled as a geometric distribution as follows:
\begin{align*}
f(x_i;p_1) &= p_1(1-p_1)^{x_i - 1} \; x_i = 1,2,... \; 0<p_1 <1 \\
f(y_i;p_2) &...
- 110k
1
vote
2
answers
61
views
Normalising likelihood for BIC/AIC calculation
I am running some model inference using AIC and BIC. My problem is that when I go and calculate the (maximum) loglikelihoods of my models, they are usually really high (range between 4700 and 1400 ...
- 5,137
13
votes
1
answer
2k
views
General theorems for consistency and asymptotic normality of maximum likelihood
I'm interested in a good reference for results concerning asymptotic properties of maximum likelihood estimators. Consider a model $\{f_n(\cdot \mid \theta): \theta \in \Theta, n \in \mathbb N\}$ ...
- 1,108
1
vote
0
answers
95
views
Confidence intervals for integer parameters
I'm interested, purely out of curiosity, in what methods can be used to calculate confidence intervals for discrete integer model parameters.
As an example, consider the model (which I can flesh out ...
- 71k
1
vote
0
answers
26
views
Asymptotic normality of penalized MLE?
Let $L(\theta;X)$ denote the log-likelihood of a model and I maximize the following to estimate $\theta$,
$$
\arg\max_{\theta}L(\theta;X)+\lambda\theta^2
$$
If $\lambda$ is 0, then the asymptotic ...
- 71k
30
votes
4
answers
8k
views
Why maximum likelihood and not expected likelihood?
Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
0
votes
1
answer
36
views
Use the delta method to find confidence intervals
Given $X_1 ... X_n \sim \textrm{Exp}(\lambda)$, I found the MLE : $$\hat{\lambda} = \frac{1}{\bar{X}}$$
Now I need to find confidence intervals for: $$\eta = \lambda \cdot \log(\lambda)$$
To do so, I ...
- 66
1
vote
1
answer
78
views
Maximum of a distribution not invariant in different coordinate systems?
Let's say we have a probability distribution in x,y space:
$$
p(x, y)=\frac{1}{4 \pi} \sqrt{x^2+y^2} \exp \left(- \sqrt{x^2+y^2}\right)
$$
This can be converted into polar coordinates as:
$$
p(r, \...
- 1,725
0
votes
1
answer
68
views
MLE phi derivation
Let dataset $D = \{(x_1,y_1),...,(x_n,y_n)\}$ where $x\in\mathbb{R}^d$ and $y_i\in\{0,1\}$
There are 2 mean vectors $\mu_0,\mu_1\in\mathbb{R}^d$ that represent the means of each feature split by label....
- 1,509
4
votes
2
answers
336
views
Understanding maximum likelihood estimation
I am told
the method of maximum likelihood says we should use the model that
assigns the greatest probability to the data we have observed;
formally, the maximum likelihood estimator is found by ...
- 91
3
votes
1
answer
6k
views
Variance of the $\hat{\sigma}^2$ of a Maximum Likelihood estimator
Given some normally distributed observations $x_1,x_2,...,x_n$
$\forall i\ x_i\sim\mathcal{N}(\mu, \sigma^2)$
the ML estimator decides that the variance that maximizes the likelihood function is (see ...
- 10.4k
2
votes
1
answer
576
views
Is a maximum likelihood estimator in an exponential family always sufficient?
An exponential family (under natural parameterization) is such that $p(X|\eta)=h(X)\exp\{\eta^\top T(X)-A(\eta)\}$, where $X$ is the data, $\eta$ is the natural parameter, and $h,T,A$ are some ...
- 5,137
3
votes
1
answer
420
views
MLE estimation of Autoregressive Conditional Poisson model
The density of an Autoregressive Conditional Poisson ACP(p,q) model is defined as
$$ f(x | \lambda_{t}) = \frac{\lambda_{t}^{x}\exp[-\lambda_{t}]}{x!},$$
where
$$\lambda_{t} = \omega + \sum_{j = 1}...
CommunityBot
- 1
1
vote
0
answers
43
views
The confusing derivation in the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy
In the section 15.5 of the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy, it discusses the Gaussian Process Latent Variable Model. The log-likelihood objective function is ...
- 11
6
votes
1
answer
674
views
In nonlinear regression, when is MLE equivalent to least squares regression?
I recently received this one line question in a job interview and was a little stumped by it.
In nonlinear regression, when is Maximum Likelihood Estimation equivalent to least squares?
- 110k
1
vote
2
answers
119
views
How to calculate the conditional maximum likelihood of independent negative binomial variables conditioned on the likelihood of the sum?
In the paper Small-sample estimation of negative binomial dispersion, with applications to SAGE data, section 4.1 Conditional maximum likelihood:
For RNA sequencing data, assume the counts for a ...
- 53
2
votes
1
answer
45
views
Why do we take mean of errors in linear regression?
I was reading about the probabilistic interpretation of linear regression and the following formula is derived using maximum likelihood estimates :
$$
\begin{align*}
β=\underset{β}{\text{argmin}}\sum_{...
- 46.6k
2
votes
1
answer
223
views
Maximum Likelihood Stopping Tolerance
I have a large scale problem that I am training with MLE. It is taking quite a long time. I would like to set a stopping condition.
How does one set a tolerance level in MLE optimization?
absolute ...
CommunityBot
- 1
2
votes
0
answers
61
views
Problem with the Fisher information matrix in case of N measurements of two observables
Let consider two observables, $x$ and $y$. Suppose that $y$ depends on the independent variable $x$ through the model $m(x; \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ is a vector of model ...
- 21
2
votes
1
answer
50
views
Help Deriving Likelihood Term When the Target is Known Probabilistically
I am trying to model data $\{Y_t,Q_t\}_{t=1}^T$, where the model is parameterized by $\theta$. $Y_t$ is a quantity where the model prediction can be solved in closed form, $\hat{Y}_t(\theta)$, where ...
0
votes
1
answer
305
views
Maximum likelihood joint probability distribution (discrete & continuous)
I am trying to find the values $v_1$ and $v_2$ that maximizes the likelihood of some observations.
I have information about $v_1$ and $v_2$ from a set of 'experiments'. In each experiment, $v_1$ and ...
CommunityBot
- 1
0
votes
0
answers
30
views
Maximum simulated likelihood of binary logit model implementation in python
I am working through the "Microeconometrics and MATLAB" book, translating the code to Python.
Here is my implementation for estimating the parameters of a binary logit model (Colab link). I ...
- 237
0
votes
0
answers
14
views
How to solve non-identifiability problem in point estimation
I am working with a normal model $X \sim N(0, \sigma^2(\theta))$, where $\sigma^2(\theta) = \frac{1}{e}\cos^2(\theta)+e\sin^2(\theta)$. My goal is to estimate $\theta$ within the range $[0, 2\pi]$.
My ...
- 63
10
votes
2
answers
468
views
Is quantile regression a maximum likelihood method?
Quantile regression allows to estimate a conditional quantile for y (like e.g. the median of y,...) from data x.
I do not see any distributional assumptions about y being made. This seems in contrast ...
- 431
5
votes
2
answers
140
views
Asymptotics of MLE without closed form solutions
Suppose $L(\theta;X)$ denotes the likelihood of a model where $\theta$ is the parameter and $X$ is the data. $L(\theta;X)$ doesn't have a closed-form solution for MLE. I use a numerical procedure to ...
- 8,101
0
votes
0
answers
25
views
Is the EM algorithm guaranteed to converge if the log likelihood is concave
As the EM algorithm is guaranteed to increase the log likelihood at each iteration. If the log likelihood is concave is it guaranteed to converge to the maximum of the likelihood, that is will we get ...
- 55
1
vote
0
answers
13
views
If the most likely value is that which minimizes squared-error, what are the possible distributions?
Gauss uniquely characterised the 1D normal distribution by asking for a distribution that:
is symmetric
is decreasing on either side of some center point $\mu$
has the data likelihood maximized by ...
- 121
4
votes
1
answer
842
views
Likelihood of Linear Discriminant Analysis compared to logistic regression
I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable:
A Linear Discriminant Analysis model where the ...
CommunityBot
- 1
-1
votes
0
answers
38
views
What are the good starting values in this question?
since when x = 1 the mean value of y is 200, so we have $\frac{\theta_1}{\theta_2+1} \approx 200$?
and does $Y_i \sim N(\frac{\theta_1+x_i}{\theta_2 x_i},\sigma^2)$ mean var(y) is a good estimation of ...
- 411
1
vote
2
answers
352
views
mle for the binomial distributed data (number of boys in families)
For example I have following dataset of number of boys in families that have 5 kids:
0 boy - 34 (number of such families)
1 boy - 128 families
2 boys - 233 families
3 boys - 267 families
4 boys - 144 ...
- 71k
19
votes
1
answer
16k
views
Relationship between Hessian Matrix and Covariance Matrix
While I am studying Maximum Likelihood Estimation, to do inference in Maximum Likelihood Estimaion, we need to know the variance. To find out the variance, I need to know the Cramer's Rao Lower Bound, ...
- 582
40
votes
1
answer
17k
views
Maximum likelihood estimators for a truncated distribution
Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
- 306k
1
vote
1
answer
1k
views
How to calculate the Expected maximum likelihood variance and mean for gaussian?
I am familiar with the Maximum Likelihood Estimation and Gaussian distribution. I have to big question from Bishop pattern recognition 2006 book.
As it is written on page 27, It says that the maximum ...
CommunityBot
- 1
0
votes
1
answer
69
views
Estimate parameters in Brownian Motion with drift, $dX_t = \mu dt + \sigma dW_t$
Consider a Brownian Motion with drift, $X$, on the interval $[0; T]$ given by
$$dX_t = \mu dt + \sigma dW_t.$$
Suppose that the interval is split into $n$ pieces of equal size to define $\Delta:=T/n$ ...
- 10.4k
0
votes
0
answers
27
views
Optimising Box-Cox lambda analytically
I'm taking a university course in statistics where the Box-Cox transform is being discussed. As I understand it, we assume that there is some $\lambda$ that makes the sample normally distributed after ...
- 101
2
votes
1
answer
34
views
To estimate the parameters of P0, P1, P2 and P3 of a functional response equation by MLE?
this is equation of functional response following polynomial logistic regression with binomial distribution Na=N (exp p0 + p1 N + p2 N^2 + p3 N^3 ) /(1 + exp (p0 + p1 N + p2 N^2 + p3 N^3)). I would ...
- 46.6k
2
votes
1
answer
109
views
Maximum likelihood vs generalized method of moments
I am trying to understand how maximum likelihood (MLE) and generalized method of moments (GMM) are related to each other. In particular, I often see people saying that MLE can be written in terms of ...
- 195
2
votes
0
answers
29
views
Confidence interval for the quantiles given interval for parameter
Let us say we have a random sample $X_1,...,X_n\sim F(\theta)$ and we do inference over $\theta$ and give a maximum likelihood estimate $\hat{\theta}$ and a confidence interval at the $\alpha$% given ...
- 268
1
vote
0
answers
13
views
I suspect that the MLE of the scale parameter in scale families of distributions is scale equivariant. Any derivative-free proof? [closed]
Suppose $f(x)=c^{-1}f_0(x/c)$, $c,x>0$. Then the MLE $\hat c_n(X_1,...,X_n)$, satisfies$$\hat c_n(aX_1,...,aX_n)=a\hat c_n(X_1,...,X_n),$$ for all $a>0.$
- 98.1k
7
votes
2
answers
444
views
Why does the MAP differ from the MLE for the uniform prior in Laplace's Rule?
Laplace's Rule of Succession produces an estimate for the probability $p$ of a Bernoulli distribution. It starts with a $Beta(1,1)$ prior (equivalent to a uniform distribution prior on $(0,1)$), and ...
- 61k
2
votes
0
answers
58
views
Estimation of parameters for a random effect model
I want to fit a simple one-way random effect model, i.e.
\begin{equation}
y_{ij} = \mu + \alpha_i + \epsilon_{ij},
\end{equation}
where
$\mu$ denotes the mean
$\alpha_i$ the effect of the $i$-th ...
- 18.3k
7
votes
3
answers
127
views
Formal Definition of Identification
This definition of identification (the bracketed part) is confusing to me because (based on my obvious misunderstanding) it fails for probit:
Probit with 2 covariates: $f=\Theta(X_1\theta_1+X_2\...
- 110k
4
votes
1
answer
426
views
Minimising KL divergence between two distributions
Say, we want to approximate a distribution $p(x)$ with $q(x|\theta)$. We do not know the distribution $p(x)$ but we can draw samples from $p(x)$. The KL divergence between the two distributions is
$$
\...
- 5,137
0
votes
1
answer
42
views
Maximum likelihood fit of left truncated Weibull distribution
I want to fit some samples to the right tail of a Weibull distribution. To fix the notation:
the samples are $\{X_i\}_{i=1,\ldots,n}$,
all samples are greater than a fixed threshold $L$,
the $X_i$'...
- 2,617
1
vote
1
answer
157
views
MLE of the Uniform Distribution
In a uniform distribution where $0\leq X \leq \theta$, the pdf is represented as $f(X|\theta) = \frac{1}{\theta}I(0\leq X \leq \theta)$, and the likelihood is $L(\theta) = \prod\frac{1}{\theta}I(0\leq ...
- 5,137
0
votes
1
answer
54
views
Find the maximum likelihood estimator of $\theta$ with pdf $f(x)=2x/\theta^2$ [duplicate]
Let $(X_1,\dots, X_n)$ be a random sample from $X$ with pdf $f(x)=2x/\theta^2$ for $0\le x\le \theta$ where $\theta>0$. Find the maximum likelihood estimator of $\theta$.
The likelihood function ...
- 5,137
0
votes
0
answers
26
views
What is the influence of multicolinearity on the likelihood ratio test?
I learned the two concepts separately and I try to find out how these concepts relate to each other.
To illustrate the question introduce three models. The models establish a relation between age (AGE)...
- 71k
0
votes
0
answers
46
views
MLE from a bivariate distribution
I have a sample of size $n$ from a bivariate distribution which have joint pdf with below form
$f_X(x)f_Y(y) \left[1+\lambda\left(2F_X(x)-1\right)\left(2F_Y(y)-1\...
- 17.1k
1
vote
1
answer
40
views
Estimating Markov chain transition probabilities from data
In a discrete time and space Markov chain, I know the formula to estimate the transition probabilities $$p_{ij} = \frac{n_{ij}}{\sum_{j \in S} n_{ij}}$$ I'm not sure however how you can find this is ...
- 1,303