# Tag Info

### Link between Cross-entropy and MLE

As you correctly claimed both MLE and CE give the exactly same optimal model parameters (at least for iid cases), there's no theoretical advantage of either objective to learn the usual point estimate ...
• 335

### OLS vs MLE when errors are not normally distributed (Laplace distributed)

OLS is BLUE regardless of the distribution of the errors, as long as they have a finite variance. However, the "Best" in BLUE refers to a specific criterion: variance, or, equivalently in ...
• 40.4k
1 vote

### "Dominance" condition for consistency of MLE

The dominance method wouldn't work in your example. It's not the actual zeroes that are the problem; you could always take $\Theta$ to be the subset of $[a,b]$ where $f(\theta)>0$. The problem is ...
• 41.4k

### How to calculate the Expected maximum likelihood variance and mean for gaussian?

This is a late answer, but I was just trying to show the same thing, so here it is. It's quite similar to the derivation suggested by the other answer (Wikipedia), but I found this easier to ...

### "Dominance" condition for consistency of MLE

It's an integrable with respect to the measure f(x|theta), so if f(x|theta) = 0, then that point won't contribute to the integral at all is my understanding. So f(x|theta) can take the value 0.
Accepted

### ML V REML for Hypothesis Testing

In the comments section of the first answer you can read that regarding the recommendation for using REML for model comparison, "your nested models need to have the same variables with fixed ...
• 26.1k
1 vote

### Why in Box-Cox method we try to make x and y normally distributed, but that's not an assumption for linear regression?

The very purpose to check if a given data is normal, if we use t-tests or analysis of variance to find significant difference between groups of data. If we apply t-tests or ANOVA without knowing if ...

### In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?

Both are correct. The $Q(Z)$ in the denominator of the first expression does not depend on $\theta$, and thus can be discarded from the optimization problem ($argmax_{\theta}$), hence obtaining the ...
• 2,636
1 vote

### When can I substitute an inverse with a pseudo-inverse in an estimator?

Yes, this might be possible. Say your vector parameter is $\beta$ and your interest is in some component (or linear function) $\alpha =a^T \beta$. Let the covariance matrix of $\hat{\beta}$ be $C$, ...
• 81.3k

### Confusion over Fisher-scoring algorithm

TLDR: always use line-search gradient descent or the BFGS algorithm to find the MLE. Fisher scoring is a bad idea. To discuss this method, we need to compare it to other methods to find the MLE. Let's ...
• 2,461
Accepted

### Convergence of MLE for non-IID data

It will depend, but there are some things that can usefully be said, especially if this is a smooth parametric model (as seems to be implied) If you have a law of large numbers and central limit ...
• 41.4k
Accepted

### Confusion over Fisher-scoring algorithm

Yes, but that matters less than you might think For canonical-link generalised linear models, which are a very popular special case, the algorithm is exactly Newton-Raphson For regression models more ...
• 41.4k

### Examples of distribution for which first-order condition is not enough for MLE

Here is the likelihood function for fitting a Cauchy distribution with scale $\gamma = 1$ and unknown location $\lambda$, where we made three observations $x_1,x_2,x_3 = 0, 6,6$. You can see that ...
• 82.1k

### Examples of distribution for which first-order condition is not enough for MLE

Two quick examples that aren't contrived: When $\langle X_i\rangle_{i\in\{1,~\ldots~,~n\}}\sim\mathrm U[0, \theta),~\theta> 0.$ It's an easy exercise to check the $n$th–order statistic is the mle ...
• 9,427

### Why do we care if the likelihood function is tractable?

It is possible to handle an "intractable" likelihood function through a range of methods. You have mentioned some of the methods, but another common one is the EM algorithm. As you point ...
• 128k

### Are least squares equivalent to ML normal distribution for any $f$?

Least Squares Estimation (LS) The least squares estimation aims to minimize the sum of squared residuals between the observed values $y_i$ and the model predictions $f(x_i, \beta)$:  LS = \arg\...
• 63.9k
The answer to your question Given $\lambda$, we can calculate $\mu_x, \mu_y, \sigma_x, \sigma_y$ using standard techniques, which you can find by searching "Weighted Least Squares" and ...