Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

Filter by
Sorted by
Tagged with
1
vote
1answer
24 views

Why are the MLE and MMSE corrections for sample variances different?

I have a number of samples of sample size 2 and a number of sample of sample size 3. If my samples are all samples from populations with a shared population variances, I wish to estimate population ...
1
vote
0answers
22 views

Bayesian update vs optimization

Say I have a multivariate normal vector $$ r \sim N(\mu , \Sigma ) \Rightarrow Pr \sim N(P\mu , P'\Sigma P ) $$ and I observe that $$ Pr = Q $$ Now I can use Bayes rule to calculate the ...
1
vote
1answer
29 views

Why do we sometimes define likelihood as $p(\textbf{T}|\textbf{X},w)$ and sometimes as $p(\textbf{X}, \textbf{T}|w)$?

Let's suppose we have a dataset $\mathcal{D}=(\textbf{X}, \textbf{T})$ where $\textbf{X}$ are the samples and $\textbf{T}$ are the targets. We want to find $w$ such that the likelihood is maximized. ...
0
votes
1answer
17 views

Logistic regression does not seem to maximize model accuracy

I'm using gradient descent to train my logistic regression model for a classification task. However, I notice that the accuracy of my model (using a boundary threshold of 0.5 to classify each sample) ...
0
votes
0answers
16 views

Estimate mean of Poisson from binary data

If you assume that counts in sample units would be distributed according to a Poisson distribution, but the data that you have are observations of only presence (count would be 1 or more) or absence (...
1
vote
0answers
16 views

Consistency of the MLE

I have a sample of size n from the following distribution: $$\frac{\alpha x^{\alpha-1}}{\beta^\alpha}$$ for $0<x<\beta$ and $\alpha > 0$. I found that the MLEs are $$\hat{\beta}=x_{(n)}$$ ...
4
votes
1answer
361 views

Fisher information for MLE with constraint

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should ...
2
votes
0answers
14 views

Can the GAN objective function be written as related to a log-likelihood of some “classical” statistical model?

Can the objective function that GAN (Generative Adversarial Network) models optimize be written as a lower bound of the log-likelihood of some "classical" statistical model? I am reading through the ...
12
votes
5answers
18k views

MLE vs MAP estimation, when to use which?

MLE = Maximum Likelihood Estimation MAP = Maximum a posteriori MLE is intuitive/naive in that it starts only with the probability of observation given the parameter (i.e. the likelihood function) ...
6
votes
1answer
2k views

Relation between MAP, EM, and MLE

I am a beginner in machine learning. I can do programming fine but the theory confuses me a lot of the times. What is the relation between Maximum Likelihood Estimation (MLE), Maximum A posteriori (...
2
votes
1answer
59 views

Asymtotic distribution of the MLE of a Uniform

A property of the Maximum Likelihood Estimator is, that it asymptotically follows a normal distribution if the solution is unique. In case of a continuous Uniform distribution the Maximum Likelihood ...
0
votes
0answers
16 views

Differences between optimization methods [on hold]

We know that the log likelihood function of a student t distribution is $logL(\nu,y)= n\times log\Gamma((\nu+1)/2))-n\times log(\sqrt{(\nu\pi)})-n\times log\Gamma(\nu/2)-\sum_{i=1}^{n}((\nu+1)/2)\...
0
votes
3answers
124 views

Is MLE of the mean of a distribution always the sample average?

From the connotation of "Maximum likelihood estimator" I am inclined to think that the maximum likelihood estimator of the mean of a distribution should equal the mean of the sample values drawn from ...
0
votes
1answer
191 views

Confusion about the use of the MLE & the posterior in parameter estimation for logistic regression

In classification one usually computes $$ C = \operatorname*{argmax}_k p(C=k\mid X) $$ where $p(C=k\mid X)$ is the posterior distribution. In a simple logistic regression setting with $C \in \{0, 1\}...
0
votes
1answer
59 views

How to find manually the value of the likelihood function?

I have a statistic homework and this is my question: Outcome (binary)=f(age, number of books) And that I have four observations in my dataset: Observation 1: Outcome=1, age=0.5, number of books=5 ...
0
votes
0answers
14 views

What might cause a high $\chi^2$ value of likelihood ratio criterion, even though transformed plot looks very similar to the original?

The label with "_norm" is the azimuthal angle $\phi$ distribution of the parameter, after Yeo Johnson transformation. I have been calculating the likelihood coefficient using $\chi^2 = 2( logL(\hat{\...
0
votes
0answers
7 views

Question about marginalization

This is related to a research project I'm working on, I hope someone can help me clear up this confusion... I have a 2d array of log-likelihood values that I obtain after sampling from my posterior ...
0
votes
1answer
9 views

classifying of naïve Bayes classifiers is to choose a $y_k$ to maximize the multiplying (joint probability), is my understanding correct?

this CMU Machine Learning Course says (naïve Bayes classifiers) classifying is just a matter of multiplying together those selected parameter estimates that happen to match the values of my new ...
9
votes
1answer
950 views

the relationship between maximizing the likelihood and minimizing the cross-entropy

There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?
2
votes
1answer
4k views

The loss function of the Gaussian maximum likelihood estimator

What is the loss function of the Gaussian maximum likelihood estimator in the classical linear regression model? I see a question asking this, but it seems that we never mention loss function when ...
2
votes
1answer
62 views

MLE as an expectation over the empirical distribution

I am reading Ian Goodfellow "Deep Learning" book. At page 128, it writes the maximum log-likelihood estimator and then says it is equivalent to the expectation over the empirical distribution To ...
3
votes
1answer
126 views

Constructing likelihood ratio test

Let $y_1$ and $y_2$ be independent Poissons with means $\lambda_1$ and $\lambda_2$. Find the likelihood ratio statistic for the test $H_0: \lambda_1 = \lambda_2$. Specify the asymptotic null ...
47
votes
4answers
63k views

Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
0
votes
0answers
17 views

Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution

I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical ...
2
votes
1answer
132 views

Do tree based methods like random forest and gradient boosting produce unbiased estimates?

Could anyone point me to literature that discuss properties of tree based estimators? For example, are they unbiased, consistent, maximum likelihood, efficient, etc?
5
votes
3answers
85 views

How to integrate the marginal likelihood numerically?

Consider a log-likelihood function $\ell(\theta,b)$, where $b\sim F$. I want to calculate the marginal log-likelihood $$\ell(\theta) = \int\exp\left(\ell(\theta,b) \right) dF(b).$$ However, $\exp(\...
0
votes
0answers
28 views

Negative Variance from inverse Hessian [on hold]

In python, I'm using the inverse Hessian as an estimator of error for parameter estimation which works fine for most trials. For some small number of the trials I get negative variances on the ...
3
votes
0answers
60 views

Maximum Likelihood - Normal Errors - When is the Jacobian needed?

I am considering the following non-linear model $$h(z) - \lambda_0 - \lambda_1 z - \lambda_2x = v$$ where $v \sim \mathcal N(0,\sigma^2)$ unobserved error and where $\lambda_j$ are unknown ...
1
vote
1answer
35 views

logistic mcmc start [closed]

I am doing logistic regression with MCMC . What is a good start for MCMC ?
0
votes
1answer
39 views

uniform distribution MLE with U($\theta$,$\theta+1$)

If my data $X_1, X_2,....,X_{10}$ has distribution of $U(\theta, \theta+1)$, and $X_{(10)} = 2$, would the MLE for $\theta$ actually be any $\theta$ in it's parameter space? I'm assuming this is ...
1
vote
0answers
40 views

Given joint density function, derive $\alpha$ and calculate maximum likelihood formula

Given the join density function $$ f(X_1 = x_1, X_2 = x_2, X_3 = x_3) = \alpha\cdot exp[{\eta _1 x_1+ \eta _2x_2 + \eta_3x_3 - w_{12}x_1x_2- w_{13}x_1x_3- w_{23}x_2x_3]} $$, where the parameter set $\...
0
votes
0answers
26 views

Evaluating Likelihood in Bootstrap Particle Filter

I am currently struggling with an attempt to apply a bootstrap particle filter to a linear, Gaussian state-space model $$s_t=A\,s_{t-1}+B\,\nu_t\qquad\text{( transition equation )}$$ $$\qquad z_t=C\,...
7
votes
3answers
662 views

Consistent unbiased estimator for the location parameter of $\mathcal{Cauchy} (\theta, 1)$

Given Cauchy distribution with pdf $p(x) = \frac{1}{\pi ((x - \theta)^2 + 1)}$ how can I find a consistent unbiased estimator for $\theta$? My reasoning so far Tried MLE, but there seems to be no ...
4
votes
1answer
2k views

4 cases of Maximum Likelihood Estimation of Gaussian distribution parameters

Let $x_1,x_2,...,x_n$ some normally distributed observations. So $\vec{x}=\begin{bmatrix}x_1 & x_2 & ... & x_n\end{bmatrix}^{T}$ In the context of my research I am trying to estimate (...
0
votes
0answers
11 views

How to determine likelihood of each observation from a fitted model in R?

Suppose you have the following data and model: ...
0
votes
0answers
3 views

Creating DLM regression in R: Error code for MLE coefficent estimation ( error code 2 from Lapack routine dgesdd) [migrated]

I am trying to implement a DLM regression model with the form: $y_t = α_t + β_{1,t}\ x_{1,t} + β_{2,t}\ x_{2,t} + v_t,\ with\ v_t ∼ N(0, σ^2)$ I used the CAPM example from Petris et. al. (2009) and ...
1
vote
0answers
28 views

GARCH modelling by maximum likelihood in R

I would like to estimate the following ARCH model in R by maximum likelihood. This question is related to: Maximum likelihood estimation of GARCH modelling in R The answer given to another question ...
10
votes
2answers
4k views

What is the Method of Moments and how is it different from MLE?

In general it seems like the method of moments is just matching the observed sample mean, or variance to the theoretical moments to get parameter estimates. This is often the same as MLE for ...
0
votes
0answers
12 views

Computing uncertainty for a non-gaussian likelihood

I have a 2 parameter model and perform a best fit on data. To depict regions of 68% confidence, I (numerically) integrate over the posterior. From this, I get a contour representing a 68% confeidence ...
0
votes
0answers
24 views

Maximum Likelihood with Experimental Data: Standard Errors and Standard Deviations

Suppose we have a set of experimental data $\{(x_i, Y_i, S_i)\}_{i=1}^N$ where the $x_i$'s are our measurement points, the $Y_i$'s are the mean value of the response $y$ over $m$ experiments at $x_i$, ...
3
votes
1answer
640 views

Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$ \theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)}) $$ ...
1
vote
0answers
18 views

How to compute statistical significance using a likelihood ratio?

The title really says it all. Suppose I have a change in log-likelihood (i.e., $\Delta LL = LL_{fitted} - LL_{null}$), and I would like to compute the $1\sigma$, $2\sigma$, etc. confidence region from ...
0
votes
0answers
21 views

Likelihood Ratio to Test Statistic of Linear Regression through the Origin

Problem I need to prove that the likelihood-ratio test can be based on the rejection region $|t|>k$ with test statistic $$ t=\frac{\hat{\beta}-k}{\sqrt{\frac{\sum_{i=1}^n(Z_i-\hat{\beta}w_i)/(n-1)}...
2
votes
0answers
27 views

Inadmissible MLE

Let us say, I have a distribution with some unknown parameter and I find the MLE of the parameter. I prove that the MLE is inadmissible using some other estimator. Why is this not in a contradiction ...
11
votes
4answers
5k views

Why are the Least-Squares and Maximum-Likelihood methods of regression not equivalent when the errors are not normally distributed?

Title says it all. I understand that the Least-Squares and Maximum-Likelihood will give the same result for regression coefficients if the model's errors are normally distributed. But, what happens if ...
1
vote
2answers
64 views

Maximum likelihood estimator when $\sum_{j} \theta_j = 1$. How to impose this condition? [duplicate]

I have a sample $x_1,\dots,x_n \stackrel{iid}{\sim}f(;\boldsymbol{\theta})$, where $\boldsymbol{\theta} = (\theta_1,\dots,\theta_d)$, and , $0<\theta_j<1$, $\sum_{j=1}^d\theta_j = 1$. I can ...
1
vote
1answer
147 views

Deriving the sampling distribution of MLE for Normal distribution

Let $X_1,\ldots,X_n$ be an observed random sample from $N_p(\mu, \Sigma)$. I know that the MLE of $\Sigma$ is $\frac{1}{n} \sum_i^n(X_i -\bar X)(X_i -\bar X)^T$, which is biased. We define $S = \...
0
votes
1answer
44 views

likelihood function of a line regression equation

I am a bit confused at how can i find the likelihood function and the solutions of likelihood function for a line equation, for example y=3x+15
0
votes
2answers
129 views

Looking for an iterative method to fit a beta-exponential distribution to a dataset

I have a messy beta-exponential distribution that has 3 variables that I have to fit to from a dataset with 50 observations. The problem is that I only know how to use Newton-Raphson for 1 variable. ...