New answers tagged maximum-likelihood
1
ERM is equivalent to MLE when the risk is defined as the negative of the likelihood. Other than that I think the differences are subtle and differ from author to author. I'd expect that usually we would only call a method MLE if it optimizes some likelihood function by selecting $\eta(x) = \mathbb{P}(Y | X=x)$ while empirical risk minimization (in the ...
2
These are different methods to estimate parameters, however they are related. The Gaussian (normal) distribution in particular has $(x-a)^2$ as a term in the loglikelihood, which means that maximising the likelihood over $a$ for independent observations (involving a product of the densities that becomes a sum after taking the log) amounts to minimising the ...
2
When the distribution (family) is known, and the sample size is large, it is generally difficult to find something better than maximum likelihood. Anyhow, to give more specific advice than that we would need to know the specifics.
It is unclear why you think binning is more robust, generally it is just information loss. But you can still use maximum ...
16
As I'm sure you know (or can easily derive), the maximum likelihood estimator of $(\mu,\sigma^2)$ is
$$(\hat\mu,\hat\sigma^2) = \left(\bar X, \operatorname{Var}(X)\right)$$
where, as usual, $n\bar X = X_1+X_2+\cdots + X_n$ and, a little unusually,
$$\operatorname{Var}(X) = \frac{1}{n}\left((X_1-\bar X)^2 + (X_2-\bar X)^2 + \cdots + (X_n-\bar X)^2\right)$$
(...
7
Assuming $q=1-p$ as usual, by the invariance property, the MLE of $g(p)=p^2(1-p)^3$ is $$\widehat{g(p)}_{ML}=g(\hat{p}_{ML})$$
So, you'll find the ML estimate for $\hat{p}$ and apply the function $g(.)$.
1
This does not seem to be the same as what is usually called "least squares fit": you compute the LSQ between a parametric model and another already fitted model: the kernel density estimator (assuming that this is what you mean with "KDE"). Note that the KDE depends on the bandwidth, and thus your method is not even well defined.
The MLE, ...
1
The strict answer is ‘yes’, as the estimation procedures are totally different, so ‘yes there is a difference between least squares and MLE’. The less strict answer is yes, the fits will likely differ by some degree, which may be large or small.
You have framed the problem to try to eliminate the difference between two empirical distributions, using a ...
1
No, there is no problem having the estimator and the parameter (estimate) in the same formula. The estimator $\hat{\theta}_{\text{MLE}}$ is the stochastic variable, the argument of the density function, while the parameter is an unknown constant, yes, the parameter of the distribution. So, what you have is correct.
2
If we consider a continuous relaxation of Bernoulli that allows the true probability to be between 0 and 1, a recent paper argues [1] that, no, cross-entropy is not adequate for $y \in [0,1]$, because it is not a Bernoulli distributed variable.
While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the ...
2
Soft labels define a 'true' target distribution over class labels for each data point. As I described previously, a probabilistic classifier can be fit by minimizing the cross entropy between the target distribution and the predicted distribution. In this context, minimizing the cross entropy is equivalent to minimizing the KL divergence. So, what we're ...
2
Here are some thoughts about your question:
The classical way to assess the quality of maximum likelihood estimators is indeed to:
generate $n$ independent and similar in size synthetic data sets from your model (parametrized with the ground truth parameters $p_1,\dots,p_m$);
compute maximum likelihood estimators for each of these data sets $({p}^{i}_1,\...
1
The likelihood being continuous over a compact parameter space is a sufficient but not necessary condition for the MLE to exist. This means that whenever you have a likelihood that is continuous and a parameter space that is compact, you know for sure that a MLE exists - but the MLE can also exist in other cases. For instance, if the likelihood is twice ...
0
The full likelihood, based on the complete data, is
$$
L(\mu)=(\frac12)^a \mu^b (3\mu)^c (\frac12-4\mu)^d
$$
But you only know $a+b$, then we make a likelihood using that $P(a ~\text{or}~ b)= \frac12+\mu$, giving the observed-data likelihood
$$
L^*=(\frac12+\mu)^{a+b} (3\mu)^c (\frac12-4\mu)^d.
$$
1
Existence a.s.
As @whuber says, the score
$$
S_n(\theta) = \frac1n\sum_{i=1}^n \frac{\partial}{\partial \theta} \log f(x_i, \theta) = -\frac{1}{\theta^2} - \frac1n\sum_{i=1}^n x_i^{\theta} \log x_i + \frac1n\sum_{i=1}^n \log x_i
$$
is a monotonically decreasing function (compute its derivative) such that
$$
\lim_{\theta \rightarrow 0^+} S_n(\theta) = \infty \...
0
You need to specify a prior for the "p". For binomial distribution, the conjugate and popular prior for "p" is beta distribution. You can then apply Bayes theorem to get the posterior distribution of p, given n and k, and also conditional expectation.
1
I would say that the methods using 'maximum likelihood' and 'iterated reweighted least squares' are independent concepts. It is only in a specific case, the generalized linear model (GLM), that the two coincide.
The article speaks of two different type of derivations.
One is the derivation done in the 70s by Nelder and Wedderburn, and in different forms by ...
3
I don't think there is an elementary derivation of the exact bias, but let's see how far we can get.
Let's start with the mean, which I'll call $\mu$, and which is equal to $1/\lambda$. The mean is estimated by the sample average
$$\hat\mu = \bar X_N =\frac{1}{N}\sum_{i=1}^N x_i$$
The sample average is unbiased for the mean, for any distribution, so we know ...
2
We don't generally consider $\lambda$ as a parameter in the model you want to estimate. It doesn't have an interpretation outside of the model, in terms of your actual data. Instead, we consider $\lambda$ as a tuning parameter or a hyperparameter. This terminology means that $\lambda$ affects how you estimate $\beta$, but you aren't interested in $\lambda$ ...
1
You're correct, the Newton-Raphson method could be used here as well. In fact, you can treat it as a root-finding method applied to the (11.78).
The trigamma function does not introduce challenge either. Surely it's not available in a closed form, but so does digamma, and there're efficient numerical procedures to compute both of them, in fact scipy has a ...
0
The log of Gaussian distribution over $\mu$ of standard deviation $\sigma_\mu$ can be written as $N(\mu,\sigma_\mu)$ and its second derivative respect to $\mu$ is $-1/\sigma_\mu^2$
when $\mu_m = \bar{x}$, $P = N(\mu_m,\sigma_\mu)$ (Proven by equation 22.6)
Therefore, comparing those two curvatures, we have
$$
\begin{aligned}
\frac {\partial^2}{\partial\mu^2}...
2
I think I got the intuition. I understood after reading the Variational inference part of the Approximate Inference chapter in the book and a section in the Wikipedia article of EM algorithm. I have replaced the $\sum$ with $\int$, so this holds for continuous Z as well. Here it goes.
We can write $p(X|θ)$ as $p(X|θ) = \frac{p(X,Z|θ)}{p(Z|X,θ)} = \frac{p(X,...
1
(a) The maximum likelihood estimation maximizes the sum of squared residuals.
False. Maximum likelihood estimation maximizes the sample likelihood function.
(b) Maximum likelihood estimation always yields a unique solution (i.e., parameter estimates).
False. Maximum likelihood estimation only guarantees a unique global maximum when the sample likelihood ...
Top 50 recent answers are included
