New answers tagged maximum-likelihood
1
If you allow me, for a moment forget about the frequentist v/s Bayesian style of things and let us simply dive into what are we "modeling".
Imagine that all samples $x$ in a dataset belong to some set $\mathcal{X}$ and the probability of observing a sample $X = x$ is $p(X = x)$. (Note the usage of capital letter for a random variable and $x$ a realization ...
0
This is functional invariance, and the transformation doesn't have to be one-to-one. So, $\hat{h}(\mu)=\bar{y}^2$. If you take the expectation and do the algebra it won't be equal to $\mu^2$, so will be biased.
4
The prior which is uniform over $[m,\infty)$ is improper, but informative: it contains the information that the value is at least $m$.
6
...example where, with an improper prior, the bayesian estimator
equals the maximum likelihood estimator...
There is a fundamental issue with this property, namely that it depends on the parameterisation of the sampling model. Indeed, if
$$\hat\theta^\text{MAP}=\arg\max_\theta L(\theta|x)\pi(\theta)=\arg\max_\theta L(\theta|x)=\hat\theta^\text{MLE}\tag{1}$...
0
Complicated likelihood or not, I suggest taking advantage of auto-differentiation tools to compute Jacobians and hessians.
In Python, there is the autograd library. This will give you exact hessians, and jacobian vectors for better convergence. See my notes here on using autograd for MLEs: https://github.com/CamDavidsonPilon/PyDataNY_2019_tutorial/tree/...
0
The occurrences of suprema (instead of maxima, which might not exist) are troublesome. Let us therefore isolate the basic underlying idea and rigorously establish it.
Definitions
Suppose $f:\Theta\to\mathbb{R}$ is any real-valued function on a set $\Theta.$ By definition, its supremum is the least upper bound of the values of $f:$
$$\sup_{\theta\in\...
2
Your two log-likelihoods are equivalent, see What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?. Another way of saying that is that log-likelihoods are only defined up to an additive constant. Here constant means a term not depending on unknown parameters (in your example $\sigma^2$.)
By ...
2
Thanks for the help Jeremy. I found the solution in Bollen (1989), p. 111.
Assume again the three-wave version of the model above, $x_{it} = \alpha_i + \delta_{it}$ and, for the sake of demonstration, assume the observed covariance matrix $S$ is
$\begin{align}
S & =
\begin{bmatrix}
x_{1}^{2} & & \\
x_{2}x_{1} & x_{2}^{2} & \\
x_{3}...
5
A function having a negative derivative over an interval implies that the function is decreasing over that particular interval.
Given that $x=0$
$$\ell'(p) = -\frac{n}{1-p} < 0 \quad \text{ for all } 0 \leq p \leq 1.$$
Therefore, $\ell(p)$ is decreasing function. Furthermore, the maximum of $\ell(p)$ must occur at $p = 0$.
1
The likelihood is missing the term $\alpha-1$, because the gamma PDF contains $x^{\alpha-1}$:
$$-\log L(\theta ) \propto -(\alpha-1)\sum\limits_{i = 1}^N {\log {({y_i} - \theta {x_i})} } + b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )$$
Minimizing this expression is a convex optimization problem with the constraints $y_i-\theta x_i>0$. The second part ...
2
Yes, it is biased because, unlike the case of the OLS, the ML variance estimator does not include a correction. That is, the correction in the denominator should be n - k where n is your sample size, and k is the number of independent variables.
Note that although the ML estimator of variance is biased, it is usually only the issue in smaller samples, where ...
1
Your model isn't clear to me, that's not necessarily bad, but I think you can express it more simply.
You don't need $B$ and $\Psi$ or $\alpha$ in your equations. These aren't necessary for the simple model you specify. You can ignore $\delta$ for this model, if you want to keep things simple and just standardize everything.
If you do this, then the ...
1
I'm sorry to have given you the wrong impression. The optimal value of $\lambda$ obtained through cross-validation increases roughly with $\frac{p}{n}$. In the example of the linked question, the change is so large that you can expect $\lambda_\text{CV}$ to decrease. However, the rough part was perhaps understated in my answer there. The optimal penalty ...
1
Let $W$ and $U$ denote two continuous (for simplicity) r.v.s with densities $f_W(w)$ and $f_U(u)$. The expected value of the r.v. $Z:=f_W(U)/f_U(U)$ is
\begin{eqnarray*}
E(Z)&=&\int_{S_U}\frac{f_W(u)}{f_U(u)}rd F_U(u)\\
&=&\int_{S_U}f_W(u)d u\\
&\leqslant&1,
\end{eqnarray*}
where $S_U$ is the support of $f_U(u)$. Then, we use Jensen'...
0
I couldn't follow his solution method but here is another way of getting the same result:
Let $ L(\mu) $ be the log likelihood as a function of $\mu$. (See equation 22.4 in the David MacKay book .)
Further let $\mu_{m}$ be the MLE, which is also the sample mean (See equation 22.6) of the N iid samples from $ \mathcal{N} (\mu,\sigma)$ where $\sigma$ is known. ...
1
Let us consider the example, suppose we are testing $$H_0: p = 0.5\\H_1: p<0.5$$ and suppose that $\bar{x} = 0.4$. $$L(X, p) = p^{\sum X_i}(1-p)^{n - \sum X_i}$$ maximizing it wrt $p$ subject to $p \le 0.5$ yileds $\hat{p} = \bar{x} < 0.5$ and hence $$\Lambda = \frac{0.4^{\sum X_i}0.6^{n - \sum X_i}}{0.5^n} \ge k \iff \sum X_i \ln 0.4 + (n - \sum X_i)\...
5
This post was brought to my attention just a few days ago. Thank you for your interest.
Question 1: What useful statistical inferences can be made using a model that makes no assumptions at all?
Before I answer this question we should agree on a definition of the word model:
A common definition of a statistical model is the set of possible probability ...
3
The contribution to your log-likelihood function due to the truncation should be $\log P_X(\tau_i;\alpha,\beta)$ not $\log 1 - P_X(\tau_i;\alpha,\beta)$. Thus, I think you just need to change lower.tail = FALSE to lower.tail = TRUE in your llik function.
Also, you are conflating censoring with truncation. From Wikipedia,
Truncation is similar to but ...
1
Let's start out with the log of the likelihood function for a single individual who quit in year $t$:
$$\mathcal{L}(\theta) = \log \theta + (t-1)\log (1-\theta)$$
If I have $Y_t$ such individuals, the log likelihood is the sum of the log likelihoods for each individual, which, since $Y_t$ represents the number of individuals who quit in year $t$, is just $...
0
The objective is to estimate the parameters or, more precisely, to get a method for their estimation (since the same form of likelihood can be applied to different data sets).
There are different ways to choose parameter estimators - maximum likelihood is just one of them, which uses as the criteria for choosing the estimator that the probability of getting ...
0
maximum likelihood can exhibit severe over-fitting for
data sets that are linearly separable. This arises because the maximum likelihood solution occurs when the hyperplane corresponding to σ = 0.5, equivalent to w^Tφ =
0, separates the two classes and the magnitude of w goes to infinity and then causes highly fluctuation in decision boundary. In this case, ...
0
If the underlying distribution has a Poisson distribution with parameter $\lambda$ and from a sample size of $n$ there are $n_0$ zeros, then the maximum likelihood estimator of $\lambda$ is $-\log(n_0/n)$.
So this is what you've posted if $P_{x=0}$ is the same thing as $n_0/n$.
An estimate of the variance is $1/n_0-1/n$.
2
Re-arranging your equation for $\hat{\alpha}$ gives the useful form:
$$\frac{1}{\hat{\alpha}} = \log(x_{(n)}) - \frac{1}{n} \sum_{i=1}^n \log(x_i).$$
This form is useful for showing the convergence of the MLE. From the laws of large numbers you are clearly going to get $x_{(n)} \rightarrow \beta$ and $\frac{1}{n} \sum_{i=1}^n \log(x_i) \rightarrow \mathbb{...
2
Regarding the distribution of $\hat\alpha$:
The MLE of $\alpha$ can be rewritten as $$\hat\alpha=\frac{n}{\sum\limits_{i=1}^n \ln \left(\frac{X_{(n)}}{X_i}\right)}$$
Now you can use the change of variables suggested by @whuber in comments, or equivalently note that whenever $X$ has the pdf in your question, $Y=\frac{\beta}{X}$ has a Pareto distribution ...
0
Do NOT take my word as gospel. That said, I think it's impossible. Or possible in a very specific way.
First problem is that D() and G() are not functions proper, they are variables. The second problem is that they are not independent. Worse, they are not only interdependent, they are recursively dependent on D-1(x) and G-1(x).
Note that the minmax on the ...
0
Bayes rule is essentially function composition of probability distribution functions. It doesn't inherently give you any parameters, it gives you a new (AKA posterior) distribution.
Now, you're trying to find a posterior that is normally distributed. In this case, it's possible, but it's not a given from Bayes alone; it just so happens that Normal Marginal +...
2
Maximum likelihood is a method for estimating parameters by maximizing the probability of the observed data. The main ingredients are:
The data: $D=(X,Y)$
The model parameters: $\theta$
The model that relates data to the parameters: $P(D|\theta)$ (which can be written differently, depending on the situation: $P(Y|X, \theta)$ is one of the possibilities.)
...
1
The first one is conditional joint, and can be written as $p(\mathbf{X},\mathbf{T}|w)=p(\mathbf{T}|\mathbf{X},w)P(\mathbf{X}|w)$. $w$ relates data to targets, so it's generally assumed that $p(\mathbf{X}|w)=p(\mathbf{X})$, which means the second term is nothing but a constant if we aim to maximize the expression with respect to $w$. So, both are valid and ...
1
Logistic regression is trained by minimizing logistic loss (or maximizing likelihood, what is equivalent), so it will be guaranteed to minimize this loss, rather then something else.
It predicts probabilities, so you cannot calculate accuracy from the outputs, unless you decide on some decision rule to make clarifications. Using $\hat y > 0.5$ is just one ...
0
Comment continued:
Here is a brief simulation with standard normal data, for which $\sigma^2 = 1.$
In the simulation v1 is a vector a million unbiased estimates $S^2,$ each
for a sample of size $n = 10.$
set.seed(1116)
m = 10^6; n = 10;
x = rnorm(m*n); DTA = matrix(x, nrow=m)
v1 = apply(DTA, 1, var) # denom n-1
mean(v1); mean((v1-1)^2)
[1] 0.9998608 # ...
6
A property of the Maximum Likelihood Estimator is, that it asymptotically follows a normal distribution if the solution is unique.
Not necessarily. So far as I am aware, all the theorems establishing the asymptotic normality of the MLE require the satisfaction of some "regularity conditions" in addition to uniqueness. Roughly speaking, these regularity ...
0
Following @ZhanXiong's Comment. Suppose we look at $n = 10^5$ samples of
size $n = 5$ from a Laplace (double exponential) population centered
at $10.$ That is, population mean and median are both 10.
The following simulation in R, illustrates that the sample means $\bar X = A$ and $\tilde X = H$ have $E(A) = E(H) = 10,$ so that both the sample mean and ...
5
A generic contradiction to your intuition is that the MLE is invariant by transformations, while the mean is not. In particular, in exponential families, the MLE is the empirical mean of the natural statistics, but not of other transforms of the sample. For instance, in a Normal $X\sim \mathcal N(\theta,1)$ sample, the MLE of $\theta$, mean of $X$, is $X$, ...
0
maximum likelihood estimator of the mean of a distribution
I don't think I've ever seen the mean be computed via MLE. Remember, MLE is about parameters, not moments of the distribution.
For a lot of distributions, the parameters just happen to be best estimated by the sample mean (see $\mu$ for the normal, $\lambda$ for the poisson), but this isn't always ...
Top 50 recent answers are included
