New answers tagged maximum-likelihood
0
You can find MLE equations for this distribution in Mahdvai and Kundu (2017) (accessible version here). As you can see from the paper, computing the MLE requires you to solve a critical point equation for $\lambda$ and you can then compute the MLE for $\alpha$ from this. The paper also contains further information on the asymptotic distribution of the MLE, ...
1
Consider the simpler case, where you have one target, $y$. The input-output relationship in neural networks is, in general,
$$y=f(\theta, x)+\epsilon$$
where, $y$ is the target, $x$ is the feature vector, $\theta$ is the set of parameters, and $\epsilon$ is the random error. It's typical to assume that the random error is distributed normally with zero-mean ...
1
When $(X_i)$ is a sequence of uniformly distributed variables $\mathcal{U}_{[0,\theta_0]}$ we can derive the distribution of the MLE $\hat \theta = \max X_i$.
We will show that $n( \theta_0 - \hat \theta)$ converge towards a non trivial (i.e. not $0$) distribution, which means that the rate of convergence of the MLE is $n^{-1}$.
First note that since $\max ...
1
You can factor out powers of 2 or 10 and count them up separately. That is, instead of
$$L(\theta)=\int\prod_{i=1}^n f(x_i|y,\theta)g(y)\,dy$$
work with, say,
$$L(\theta)=10^{-an}\int\prod_{i=1}^n 10^af(x_i|y,\theta)g(y)\,dy$$
where the exponent is chosen to keep $f(x_i|y,\theta)$ a reasonable size. Then
$$\log L(\theta)=-2n\log 10 +\log \int\prod_{i=1}^n 10^...
2
The ordinary-language and technical meanings of biased are different. The answer by @Glen_b gives a good description of why maximum likelihood estimators can easily be biased in the technical sense.
It is possible for the maximum likelihood estimator to be biased in something like the ordinary-language sense, but it's not usual. Something has to go wrong.
...
2
Let $z \sim \mathcal{N}(\mu, \sigma)$. Then
$$ \dfrac{z-\mu}{\sigma} \sim \mathcal{N}(0,1)$$
Conversely, if $x \sim \mathcal{N}(0,1)$, then
$$ \mu + \sigma x \sim \mathcal{N}(\mu,\sigma)$$
The noise is normal $e_i \sim \mathcal{N}(0,\sigma)$, so if I add some noiseless constant to this random variable, the mean changes
$$ f(x_i) + e_i = d_i \sim \mathcal{N}(...
0
Brief expansion of user Christoph Hanck's comment:
The measurements $x_i$ are assumed to be known exactly.* Under this assumption, $f(x_i)$ is distributed like a normal distribution with mean $f(x_i)$ and variance $0$. If $e_i\sim\mathcal{N}(0, \sigma^2)$, it follows that
$$\underbrace{d_i}_{\sim\mathcal{N}(f(x_i), \sigma^2)} = \underbrace{f(x_i)}_{\sim\...
1
The model is not identified, meaning there is no unique solution to the optimization problem. There are infinite values of the parameters that will yield the same likelihood. For example, $\alpha = .5$, $\beta = 1$, and $\sigma = 2$ will yield the exact same likelihood as $\alpha = 1$, $\beta = 2$, and $\sigma = \sqrt{19}$.
More generally, consider the ...
10
The most relevant reference imho is Steve Stigler's "Epic history of maximum likelihood" (2007)
"There were early intelligent comments related to this problem [of
seeking the most probable distribution for the observation] already
in the 1750s by Thomases Simpson and Bayes and by Johann Heinrich
Lambert in 1760, but the first serious assault ...
0
Consider $\log f(x) = -0.5\log (2 \pi \theta) - 0.5 \frac{(x - \theta)^2}{\theta}$ and
$$
\frac{\partial}{\partial\theta} \log f(x) \propto -\frac{1}{\theta}+\frac{x^2}{\theta^2} -1
$$
Thus,
$$
\frac{\partial}{\partial\theta} \ell (x) = 0 = -n(1 + \frac{1}{\theta}) +\sum \frac{x_k^2}{\theta^2}
$$
so $\theta^2 + \theta = \frac{1}{n}\sum x_k^2$ which gives
$\...
0
Just to record a variant on the above calculation with a couple of shortcuts:
From $\frac{x-\theta}{1+(x-\theta)^2}=\frac{x+\theta}{1+(x+\theta)^2}$ we see that the function $g(y)=\frac{y}{1+y^2}$ takes the same value at $y_1=x-\theta$ and $y_2=x+\theta$, hence so does $\frac{1}{g(y)}=y+\frac{1}{y}$. This function is 2 to 1 (except at $y=\pm 1$) so either $...
0
Here is my solution to "What is the simplest practical method to implement?" using python, specifically numpy, scipy and tick.
One modification is that I set the exponential kernel such that alpha x beta x exp (-beta (t - ti)), to coincide with how tick defines exponential kernels:
https://x-datainitiative.github.io/tick/modules/generated/tick....
1
If you have data $X_1, X_2, \dots, X_n$ randomly sampled from $\mathsf{Norm}(\mu, \sigma),$ with $\mu$ known and $\sigma$ unknown, then
$V = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2$ has $E(V) = \sigma^2,$ with
$\frac{nV}{\sigma} \sim \mathsf{Chisq}(\nu = n).$ The proof follows directly
from the definition of $\mathsf{Chisq}(\nu = n)$ as the distribution of the
...
0
I think you might be missing the fact that the expected value applies only when you do the sampling over and over, not on any one sample. On any given sample, a bad estimator (not unbiased, not MLE, high variance) might have a really good estimate, but we have no way of knowing by how much any given estimator misses, so we prove properties and take our ...
2
The equation is nothing but the definition of conditional probability. If you ignore the integrals, and just think of summation instead, it looks quite straightforward. By definition,
$P(\theta_1|Y)=P(\theta_1,Y)/P(Y)$.
Now we will show that the numerator and denominator work out exactly to those above.
The numerator is just integrating over $\theta_2$, and ...
1
A Bayesian point estimator is a summary of the posterior used for a specific purposed and justified by an optimisation principle associated with a utility function $U$. The point estimate $\hat\theta(\cdot)$ is the function that maximises
the posterior expected utility $$\mathbb E[U(\hat\theta,\theta)|x]=\int U(\hat\theta,\theta)\,\pi(\theta|x)\,\text d\...
3
The expression for the AR(2) log likelihood (eq. 5.3.8 in Hamilton) has a term for the joint density of $y_1$ and $y_2$. This term indeed only makes sense if the process is stationary as it involves the stationary variance-covariance matrix of $y_1,y_2$.
If you run unconstrained optimisation on the expression you indeed risk ending up with meaningless ...
2
Consider subsampling your observations, you dont need to use all 300,000 observations to estimate 9 parameters. That is an incredibly large amount of data for a fairly low dimensional parameter vector. If you picked a sub-sample of (say) 10,000 observations then you could minimise this partial likelihood instead which will be a lot faster, and the minimum is ...
2
Unless for trivial problems, of simple hyperparameter tuning, nobody uses grid search as an optimization algorithm. Mixture of beta distributions is parametrized by continuous parameters, so there is infinitely many values of the parameters, and their combinations, that you would need to check. This would not end in finite time if using grid search.
There is ...
1
Hi: If you have a linear model, then there is a closed form solution for the coefficients and these will be the same ones that would result if you assumed a normal distribution for the error term ( of the linear model ) and then maximized the resulting likelihood function.
In all other cases ( or atleast most that I know of ), non-linear models or non-...
4
If in reality $\beta>0$ then for large enough $n$ you will have $\hat\beta>0$ and you can use the delta method
$$\mathrm{var}[\hat\beta] = \left(\frac{d\beta}{d\gamma}\right)^2\mathrm{var}[\hat\gamma]= \beta^2\mathrm{var}[\hat\gamma]$$
In this case the transformation hasn't really gotten you much, but it's valid.
The problem comes when an unconstrained ...
0
The keyword is "estimation". You can not calculate the exact parameter of a distribution from finite observations, only an estimate of it. Under certain regularity conditions, MLE has good properties (efficiency, asymptotic normality, consistency). Therefore, if you don't know what is the "best" estimation of a parameter, you can use MLE ...
3
Very often, there are several possible (and reasonable) ways to estimate parameters from data. MLE gives us a sensible way to find one of them with some good properties.
I suppose the question comes from the fact that often the ML estimator is just the estimator anybody would have obviously used without all the trouble of finding the MLE. For example, even ...
2
The standard critical values will be too aggressive as the K-S test doesn’t take into account the sample error in the MLE estimates. You therefore need to bootstrap the critical values to form a valid inference.
4
This is an invalid procedure.
https://en.m.wikipedia.org/wiki/Kolmogorov–Smirnov_test
Scroll down to “Test with estimated parameters”. Disappointingly, they do not give much of a reference, but the book referenced in that paragraph might explain. (Cross Validated has many posts on this topic, too, though it would be nice to see it discussed in some primary ...
0
Yeah, so I think we could use the $\delta$-method here:
$$x(\beta_0, \beta_1) = \frac{\log \frac{p}{1-p} - \beta_0}{\beta_1} \\
x(\hat \beta_0, \hat \beta_1) \approx x(\beta_0, \beta_1) + \nabla x(\beta_0, \beta_1)^T(\hat{\boldsymbol \beta} - {\boldsymbol \beta}) \\ = \frac{\log \frac{p}{1-p} - \beta_0}{\beta_1} - \frac{\hat \beta_0 -\beta_0}{\beta_1} - \...
3
If you need the exact variance you're in trouble, I think. If not, progress can be made.
To start with, the monotone likelihood ratio property of exponential families means that $\hat\lambda$ is either the integer below the mean $\bar X$ or the integer above the mean. The variance of $\bar X$ is $\lambda/n$.
Consider some ranges of ($n$,$\lambda$):
If $\...
Top 50 recent answers are included
