Skip to main content

Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

Filter by
Sorted by
Tagged with
0 votes
0 answers
34 views

Link between Cross-entropy and MLE

There are numerous material that show the relationship between MLE and cross-entropy. Typically, these are the steps taken to show the relationship for a I.I.D data generating process $D = (X,Y)$: $$ ...
spie227's user avatar
  • 45
0 votes
0 answers
13 views

An example problem of converting a maximum likelihood problem into a restricted maximum likelihood problem

I have a question about this derivation. What is an example value of the actual matrix $A'$ such that $A'X=0$, $A'A=I$, and $\frac{1}{n}\Sigma((A'Y_{i}-mean(A'Y))^{2}=\frac{1}{(n-1)}\Sigma((Y_{i}-...
A Friendly Fish's user avatar
0 votes
1 answer
35 views

OLS vs MLE when errors are not normally distributed (Laplace distributed)

We say that under assumptions of the Gauss-Markov theorem, OLS is BLUE. The Gauss-Markov theorem doesn't mention the normality of errors. If the errors are distributed as per the Laplace distribution,...
ordinary least circles's user avatar
0 votes
1 answer
26 views

ML V REML for Hypothesis Testing

I am curious about doing hypothesis testing with REML values via a likelihood ratio test, or whatever test would make sense for REML values, but these two highly upvoted answers to existing cross ...
A Friendly Fish's user avatar
2 votes
1 answer
47 views

In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?

From cs 229 page 6: Intuitively, the EM algorithm alternatively updates Q and θ by a) setting Q(z) = p(z|x; θ) following Equation (8) so that ELBO(x; Q, θ) = log p(x; θ) for x and the current θ, and ...
figs_and_nuts's user avatar
0 votes
0 answers
17 views

Should one account for the known variance of fixed X when estimating its relationship with random Y?

In Aldrich (2005), and specifically in sections 10 and 11, the author describes the sufficient statistic for the parameter $\beta$ in the simple regression of random $Y$ on fixed $X$, with a bivariate ...
virtuolie's user avatar
  • 642
3 votes
2 answers
57 views

Confusion over Fisher-scoring algorithm

Given a probability model $f(X;\theta)$ and a set of i.i.d. observations $x_1,\ldots,x_n$ which we assume to be drawn from some true parameter $f(X; \theta_0)$, we can perform maximum-likelihood ...
shem's user avatar
  • 186
1 vote
1 answer
53 views

Convergence of MLE for non-IID data

Consider calculating optimal model parameter $\theta$ using MLE for the following 2 cases: Data generating process is independent but non identical: $L(y;\theta) = \prod_{i=1}^{n} f_{i}(y_i;\theta)$....
spie227's user avatar
  • 45
2 votes
0 answers
42 views

No Existence of Efficient estimator

I need to prove that given $(X_1,...,X_n)$ from the density $$\frac{1}{\theta}x^{\frac{1}{\theta}-1}1_{(0,1)}$$ no efficient estimator exists for $g(\theta)$=$\frac{1}{{\theta}+1}$. I have shown that ...
Onofrio Olivieri's user avatar
6 votes
2 answers
473 views

Examples of distribution for which first-order condition is not enough for MLE

As stated by the title, I am looking for an example (if any exists) of a distribution for which annulling the gradient of the (log-)likelihood function w.r.t. the parameters is not enough to ensure we ...
MysteryGuy's user avatar
5 votes
1 answer
219 views

Why do we care if the likelihood function is tractable?

I'm learning (deep) generative models and I've seen many places where the difficulty is that the likelihood function (with some parameters defined by our model) is intractable, i.e., it involves ...
Daniel Mendoza's user avatar
0 votes
0 answers
10 views

Deriving the log-likelihood function for ACD (Autoregressive Conditional Duration) models

I am trying to understand the procedure that is shown in this survey to obtain the log-likelihood function to estimate the parameters for an ACD model: PACURAR, Maria. Autoregressive conditional ...
Residual Claimant 's user avatar
0 votes
0 answers
14 views

Rejection region in LRT test

Let's say I have $X_i \sim Bi(1, \theta$) and want to test $H_0: \theta \geq \theta_0$ vs $H_1: \theta < \theta_0$. I've found that $\lambda = \frac{\sup_{\theta \in \Theta_0}L(\theta)}{\sup_{\...
Peter Sampodiras's user avatar
1 vote
0 answers
32 views

Small sample MLE vs OLS efficiency

MLE estimates are asymptotically efficient. Both MLE and OLS estimates are asymptotically normal and for many distributions their limiting variances coincide (information for one observation being the ...
memeplex's user avatar
  • 141
0 votes
0 answers
102 views

Maximum likelihood for time series notation

I am going over the following slides where the likelihood for time series is formulated as (page 5): $f(y) = \prod_{i=2}^{n} f_{y_i|y_{i-}}(y_i|y_{i-}) f_{y_1}(y_1)$ I am not quite sure what the ...
spie227's user avatar
  • 45
1 vote
0 answers
18 views

How FIML handles missing data

As title states, I have a question about how FIML (full information maximum likelihood) handles missing data. My understanding is that FIML only extends to missing outcomes and has limited ability to ...
KrisLJJ's user avatar
  • 33
2 votes
1 answer
121 views

Closed Form Solution for Gaussian Matrix which is Convex Combination?

I already asked a pretty similar question here, but the answer was inconclusive and now this related problem has come up again here. My problem is as follows, I have a $2n$-dimensional multivariate ...
A Friendly Fish's user avatar
0 votes
0 answers
15 views

How are the MLE/MAP distinction and the generative/discriminative distinction related?

What is the relationship between Maximum Likelihood Estimation versus Maximum A Posteriori Estimation and generative modeling versus discriminative modeling? Is MLE an example of a generative model ...
user10478's user avatar
  • 123
0 votes
0 answers
40 views

How to fit data to a parametric curve/model (x(t), y(t)?

I've got data of x and y pairs and I'd like to fit it to a model that is parametrized as f = (x(t), y(t)). Unfortunately, there is no way for me to analytically solve for t and get a direct ...
StatAnomaly123's user avatar
1 vote
0 answers
85 views

Estimation in generalized additive models

I'm currently trying to learn about generalized additive models (GAMs) with the book Generalized Additive Models An Introduction with R by Simon N. Wood. However, I have some questions regarding the ...
Dude3400's user avatar
0 votes
0 answers
32 views

What is the difference between estimating parameters via MLE versus minimizing deviations from expectation?

What is the difference between estimating parameters using MLE (or MAP with uniform priors): $$\theta^* = \arg \max_\theta p(X|\theta)$$ and estimating them according to which setting would engender ...
actinidia's user avatar
  • 145
7 votes
3 answers
242 views

biased MLEs in negative binomial models

I found the maximum likelihood parameters of negative binomial models are biased. An example code is provided below. Is this normal? Are there ways to obtain unbiased estimates? ...
quibble's user avatar
  • 1,694
0 votes
0 answers
33 views

Fitting Hypergeometric distribution requires non-integer arguments?

I have a vector (length s) of observations, x are class "0" and s-x are class "1" and are drawn from a population of size N. Hence, they follow the hypergeometric distribution: $$H(...
Jesús Castrejón's user avatar
0 votes
0 answers
29 views

Which log-likelihood is to be maximized for left-truncated count data?

What is to be done, if the count data is missing the counts on the zeros (i.e. left truncated data)? Say one wants to estimate a Poisson regression and the goal is to derive the log-likelihood to be ...
Marlon Brando's user avatar
1 vote
0 answers
28 views

Convexity of negative log-likelihood of exponential family distribution [closed]

Let $p(y; \theta^Tx) = b(y) \space \exp\big(\theta^T x y - a(\theta^T x)\big)$, where $x$ and $\theta$ are $d$-dimensional vectors and $y$ a scalar. If I'm not mistaken, the negative log-likelihood ...
Arturo Sbr's user avatar
0 votes
0 answers
24 views

Likelihood test on simple regression

I'm working through Introduction to Econometrics by Gary Koop, but I'm slightly struggling to make sense of a question from chapter 4. It asks you to carry out a liklihood ratio test of $H_0 :\beta=0$ ...
Student123's user avatar
0 votes
0 answers
20 views

Manual MLE of AR(1) yields a weird initial value $y_0$

I am playing with a manual implementation of the maximum likelihood estimator (MLE) of the parameters in an AR(1) model $$ y_t = c + \varphi_1 y_{t-1} + \varepsilon_t $$ with $\text{Var}(\varepsilon_t)...
Richard Hardy's user avatar
3 votes
1 answer
93 views

likelihood ratio tests on bounded parameters

I am confused by the likelihood ratio test's boundary condition limitation. A commonly stated is that it causes problem for variance parameter because it is bounded below by 0. Can these models ...
quibble's user avatar
  • 1,694
3 votes
0 answers
29 views

Is there a likelihood penalization or (im)proper prior to remove estimation bias for gamma parameters?

So I am learning that maximum likelihood estimation of the parameters for a gamma distribution are biased. As far as I understand there is no guarantee in general that there exists a prior (or base ...
Galen's user avatar
  • 9,361
0 votes
0 answers
32 views

MLE of marginal distribution for continuous random variable

Let $\mathcal{F}$ be a family of multivariate probability densities such that for a sufficiently large data sample, there always exists a unique MLE. Assume also that all marginal and conditional ...
12345's user avatar
  • 213
0 votes
0 answers
8 views

Precision of estimates of lower bit error probabilities at higher SNR

For my university lab in wireless communications, I simulated a simple uncoded BPSK (binary phase shift keying) channel with AWGN (additive white gaussian noise) to estimate the BER (bit error rate) ...
Scarab's user avatar
  • 1
0 votes
0 answers
16 views

Optimisation of Polynomial Fittting Process

I have built a multitvariate log link GLM model and I want to fit polynomials to some of the numerical variates (i.e. fit polynomials of order 1,2,3 etc to the relativities of the model). However, I ...
JDSH's user avatar
  • 1
0 votes
0 answers
15 views

When we talk about the Likelihood in general ... Are we talking about a "Joint Likelihood" or "Conditional Likelihood"? [duplicate]

I am confused about this. Suppose we collect some data and believe it came from a Normal Distribution. To estimate parameters (mu, sigma) of this Normal Distribution we create a Likelihood Function ...
user avatar
1 vote
0 answers
92 views

Manual maximum likelihood estimation of realized GARCH behaving poorly

I'm trying to estimate the maximum likelihood of a realized GARCH model. Below are the equations and the parameters I want to estimate I'm using the below function to maximise the likelihood, but it ...
user199's user avatar
  • 23
1 vote
0 answers
32 views

How to show that MLE of probit regression does not exist due to data separability

Claim The claim is the is the following: Assume we have the simple probit model $E(y_i|x_i ) = Φ(α+\beta x_i)$. Now suppose that $y_i = 1$ for all $x_i ≤ 10$ and $y_i = 0$ for all $x_i > 10$. Then $...
Marlon Brando's user avatar
0 votes
0 answers
83 views

What is the likelihood of a regression? [duplicate]

I understand linear regressions themselves have likelihoods. Is this simply the likelihood of the error? I thought it was the likelihood of the data for Y given X. In other words, $Lik(Y$~$X)=Lik(Y|...
A Friendly Fish's user avatar
4 votes
2 answers
121 views

Must maximum likelihood method be applied on a simple random sample or on a realisation?

I guess my trouble is not a big one but here it is: when one applies maximum likelihood, he considers the realization $(x_1, \dots, x_n)$ of a simple random sample (SRS), leading to ML Estimates. But ...
MysteryGuy's user avatar
3 votes
1 answer
36 views

describing binomial data in likelihood models

Binomial data can be described in various ways. Suppose we flip a fair coin twice and get one head (success). One method to calculate its negative log-likelihood is ...
quibble's user avatar
  • 1,694
0 votes
0 answers
47 views

MLE and UMVUE from an ordered sample of the exponential distribution

I am having a lot of trouble with every part of the problem below. Now, finding MLE's is simple in principle. I just find the distribution for $Y$ and then use calculus to find the value of $\sigma$ ...
Anon's user avatar
  • 155
6 votes
1 answer
353 views

Is OLS asymptotically the best estimator even without gaussian error?

It is known that MLE is consistent and asymptotically efficient. OLS under certain assumptions is asymptotically normal. If the errors are gaussian, then OLS is equivalent to MLE. If the errors are ...
user405777's user avatar
3 votes
0 answers
61 views

Probability that normally distributed variables will have a specific ranking

There are $k$ players playing a game, each gives a performance $X_k \sim N(\mu_k, 1)$ and we observe their ranking from best to worst (a permutation of the player indexes). How to calculate the ...
fhucho's user avatar
  • 131
2 votes
1 answer
42 views

Invariance property of MLE - Proof

I am reading a proof about the invariance property of the MLE but there is a equality that does not make sense to me. Suppose $\tau(\cdot)$ is a one to one function, define $\eta = \tau(\theta)$ so we ...
Lucas cantu's user avatar
0 votes
1 answer
21 views

Computing the limiting distribution of the Bayes estimator for exponential data with a Gamma prior (by using consistency?)

Let data be $X_i \sim \text{Exp}(\theta)$ iid, $i=1,...,n$. Let the prior be $\text{Gamma}(\alpha, \beta)$. The posterior is then of course $\text{Gamma}(\alpha + n, \beta + \sum X_i)$. The Bayes ...
Featherball's user avatar
3 votes
1 answer
115 views

why is MLE formula divided by sample size?

MLE for independent data samples $D = (x_1,y_1), (x_2,y_2), (x_3,y_3) ... (x_N, y_N)$ can be formulated as $$ L(D) = \prod_{i=1}^N p_i(x_i, y_i) $$ And the log likelihood being: $$ \log(L(D)) = \sum_{...
spie227's user avatar
  • 45
1 vote
0 answers
60 views

Maximum Likelihood in High Dimensions [closed]

What are some examples of high-dimensional random variables for which MLE are solved using numerical methods because we are unable to explicitly solve the equations nicely? The only example to comes ...
Nicolas Bourbaki's user avatar
1 vote
1 answer
35 views

Maximum Likelihood for Multivariate Regression (ML)

TL;DR: How do you perform likelihood maximization for a multivariate regression in the context of ML? Background: For univariate regression we can view datapoints $(x, y)$ as being sampled from a ...
asras's user avatar
  • 11
2 votes
1 answer
52 views

equivalence between the likelihood ratio test and t-tests

The linked sites (link1, link2) demonstrate that the likelihood ratio tests and the corresponding one- and two-sample t-tests are equivalent. However, based on my understanding, the null distribution ...
quibble's user avatar
  • 1,694
0 votes
0 answers
26 views

Can complicated $\mathcal{Q} (\theta; \theta^{\text{old}})$ function be replaced by log-likelihood when implementing/coding EM algorithm?

I am working on a MLE problem where one of the parameters does not have a closed-form solution. I have a proposal for $\theta^{\text{new}}$, but reject it if it does not improve $\mathcal{Q} (\theta; \...
Stew P. Doe's user avatar
1 vote
0 answers
7 views

Clarifying between sample date and input data

I'm reading section 2.6.3 from the book "The Elements of Statistical Learning" of "Trevor Hastie", "Robert Tibshirani" and "Jerome Friedman" and I have a few ...
alksdhalksjdb's user avatar
0 votes
0 answers
11 views

Analyze the variance of one adaptive process?

I'm currently interested in analyzing the variance of one adaptive process. To be more specific, suppose I have done some, let's say $n$ times, experiments where the results depend on some unknown ...
narip's user avatar
  • 185

1
2 3 4 5
69