Skip to main content

New answers tagged

0 votes

Link between Cross-entropy and MLE

As you correctly claimed both MLE and CE give the exactly same optimal model parameters (at least for iid cases), there's no theoretical advantage of either objective to learn the usual point estimate ...
cinch's user avatar
  • 335
0 votes

OLS vs MLE when errors are not normally distributed (Laplace distributed)

OLS is BLUE regardless of the distribution of the errors, as long as they have a finite variance. However, the "Best" in BLUE refers to a specific criterion: variance, or, equivalently in ...
jbowman's user avatar
  • 40.4k
1 vote

"Dominance" condition for consistency of MLE

The dominance method wouldn't work in your example. It's not the actual zeroes that are the problem; you could always take $\Theta$ to be the subset of $[a,b]$ where $f(\theta)>0$. The problem is ...
Thomas Lumley's user avatar
0 votes

How to calculate the Expected maximum likelihood variance and mean for gaussian?

This is a late answer, but I was just trying to show the same thing, so here it is. It's quite similar to the derivation suggested by the other answer (Wikipedia), but I found this easier to ...
Shivay Vadhera's user avatar
0 votes

"Dominance" condition for consistency of MLE

It's an integrable with respect to the measure f(x|theta), so if f(x|theta) = 0, then that point won't contribute to the integral at all is my understanding. So f(x|theta) can take the value 0.
Michael Gmeiner's user avatar
2 votes
Accepted

ML V REML for Hypothesis Testing

In the comments section of the first answer you can read that regarding the recommendation for using REML for model comparison, "your nested models need to have the same variables with fixed ...
Christian Hennig's user avatar
1 vote

Why in Box-Cox method we try to make x and y normally distributed, but that's not an assumption for linear regression?

The very purpose to check if a given data is normal, if we use t-tests or analysis of variance to find significant difference between groups of data. If we apply t-tests or ANOVA without knowing if ...
Gene Uniana's user avatar
0 votes

In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?

Both are correct. The $Q(Z)$ in the denominator of the first expression does not depend on $\theta$, and thus can be discarded from the optimization problem ($argmax_{\theta}$), hence obtaining the ...
Camille Gontier's user avatar
1 vote

When can I substitute an inverse with a pseudo-inverse in an estimator?

Yes, this might be possible. Say your vector parameter is $\beta$ and your interest is in some component (or linear function) $\alpha =a^T \beta$. Let the covariance matrix of $\hat{\beta}$ be $C$, ...
kjetil b halvorsen's user avatar
2 votes

Confusion over Fisher-scoring algorithm

TLDR: always use line-search gradient descent or the BFGS algorithm to find the MLE. Fisher scoring is a bad idea. To discuss this method, we need to compare it to other methods to find the MLE. Let's ...
Guillaume Dehaene's user avatar
3 votes
Accepted

Convergence of MLE for non-IID data

It will depend, but there are some things that can usefully be said, especially if this is a smooth parametric model (as seems to be implied) If you have a law of large numbers and central limit ...
Thomas Lumley's user avatar
2 votes
Accepted

Confusion over Fisher-scoring algorithm

Yes, but that matters less than you might think For canonical-link generalised linear models, which are a very popular special case, the algorithm is exactly Newton-Raphson For regression models more ...
Thomas Lumley's user avatar
9 votes

Examples of distribution for which first-order condition is not enough for MLE

Here is the likelihood function for fitting a Cauchy distribution with scale $\gamma = 1$ and unknown location $\lambda$, where we made three observations $x_1,x_2,x_3 = 0, 6,6$. You can see that ...
Sextus Empiricus's user avatar
6 votes

Examples of distribution for which first-order condition is not enough for MLE

Two quick examples that aren't contrived: When $\langle X_i\rangle_{i\in\{1,~\ldots~,~n\}}\sim\mathrm U[0, \theta),~\theta> 0. $ It's an easy exercise to check the $n$th–order statistic is the mle ...
User1865345's user avatar
  • 9,427
5 votes

Why do we care if the likelihood function is tractable?

It is possible to handle an "intractable" likelihood function through a range of methods. You have mentioned some of the methods, but another common one is the EM algorithm. As you point ...
Ben's user avatar
  • 128k
2 votes

Are least squares equivalent to ML normal distribution for any $f$?

Least Squares Estimation (LS) The least squares estimation aims to minimize the sum of squared residuals between the observed values $ y_i $ and the model predictions $ f(x_i, \beta) $: $$ LS = \arg\...
Robert Long's user avatar
  • 63.9k
4 votes
Accepted

Closed Form Solution for Gaussian Matrix which is Convex Combination?

The answer to your question Given $\lambda$, we can calculate $\mu_x, \mu_y, \sigma_x, \sigma_y$ using standard techniques, which you can find by searching "Weighted Least Squares" and ...
John Madden's user avatar
  • 4,550

Top 50 recent answers are included