Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

865 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
18
votes
0answers
773 views

Fitting custom distributions by MLE

My question relates to fitting custom distributions in R but I feel it has enough of a probability element to remain on CV. I have an interesting set of data which has the following characteristics: ...
11
votes
0answers
1k views

Testing for a significant difference between ML estimates: Likelihood ratio or Wald test?

I am trying to test whether or not there is a significant difference between maximum likelihood estimates of two genetic parameters (selection and dominance) across two environments with genotype data ...
10
votes
0answers
526 views

Does using bootstrapped samples improve parameter estimates for a fitted distribution?

The R package retimes has a function for fitting an ex-Gaussian distribution to a set of observations. The method involves taking multiple bootstrapped samples of the observations, and fitting the ex-...
9
votes
0answers
331 views

MLE, regularity conditions, finite and infinite parameter spaces

The problem I have is in figuring out why the MLE is no longer consistent in countable parameter spaces under conditions specified below. The set up is as follows: we are consider a parameters space ...
7
votes
0answers
230 views

asymptotic unbiasedness of weibull mle

It's known that the MLEs of the two-parameter Weibull distribution scale and shape parameters are not available in a closed form. It is, however, known that they do exist, are unique, and moreover, ...
6
votes
0answers
134 views

Bayesian inference via approximate data likelihood

Suppose that we have a very large i.i.d. sample $x_1,...,x_n$ and a data likelihood defined by $$p(x | \theta,\beta) = \prod_ip(x_i | \theta,\beta)$$. Further suppose that $\theta$ is the parameter ...
6
votes
0answers
1k views

Why is Fisher Scoring easier to compute?

In practice, the observed information matrix (Newton-Raphson) is usually replaced by its expectation, known as Fisher scoring. Link: https://en.wikipedia.org/wiki/Scoring_algorithm#Fisher_scoring ...
6
votes
0answers
666 views

Likelihood maximization: MCEM algorithm versus MCMC algorithm

Hello Everyone this is my first question. I am a particle physicist and I am doing some empirical studiues on parameters estimation using different methods (this might give me some handle to study on ...
6
votes
0answers
1k views

“weight” input in glm.nb function in R. How exactly does the weight affect the likelihood?

I would like to understand how the weight argument of glm.nb is affecting the likelihood function. I understand that glm.nb find ...
6
votes
0answers
153 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
6
votes
0answers
662 views

Computing Standard Errors in EM algorithm

I'm applying the EM to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov chain, ...
5
votes
0answers
100 views

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
5
votes
0answers
58 views

What are some of the robustness checks for the likelihood ratio test?

In the application of statistical methods in social science, one usually does a lot of robustness checks. If I got some publishable findings using LRT test by discrimination two theoretical models, ...
5
votes
0answers
133 views

Maximum likelihood and degenerate Fisher information

I am wondering if there are some standard results to find rates of convergence of the MLE for different sub-parameters, when the Fisher information is degenerate. More precisely, suppose that I ...
5
votes
0answers
1k views

Odds ratio MLE calculation: unconditional vs conditional?

I am running an odds ratio calculation for site methylation amongst cases and controls. In this situation is it preferable to use a conditional or unconditional MLE? I am asking because R uses a ...
5
votes
0answers
598 views

Implementing ARMA Log Likelihood with the Kalman Filter Algorithm

A popular algorithm to determine the (complex) log likelihood function of an ARMA(p,q) process involves generating it through the use of a state-space model and the Kalman Filter. I started reading ...
5
votes
0answers
341 views

Relationship of Poisson regression to determining proportions in a mixture distribution

I am a particle physicist, and a very frequent task is: given data sampled from a distribution mixture $$ Z \sim \sum_i \phi_i F_i, $$ where $\phi_i$ are the mixture proportion / prior probabilities ...
5
votes
0answers
294 views

Bayesian estimates for Deming regression coinciding with least-squares estimates

Consider the following Deming model with independent replicates : $$x_{i,j} \mid \theta_{i} \sim {\cal N}(\theta_{i}, \gamma_X^2), \quad y_{i,j} \mid \theta_{i} \sim {\cal N}(\alpha+\beta\theta_{i}, \...
5
votes
0answers
1k views

Parameter Estimation for Naive Bayes - Maximum a posteriori and Maximum Likelihood

I am wondering if I understand those terms correctly. To summarize my thoughts: In naive Bayes, our decision rule is basically the Maximum a posteriori (MAP) estimate of our hypothesis. We assign an ...
5
votes
0answers
510 views

Intuitive explanation of “integrate out random effect”

We are trying to figure out an intuitive reasoning behind integrate out the unobserved random effect. The specific formula is: $f\big(y_i|x_i;\beta, \sigma_c\big)=\int_{-\infty}^{+\infty}\Big(\prod_{...
5
votes
0answers
2k views

Frequency weights, rare events and logistic regression

I'm working on a model that requires me to look for predictors for a rare event (less than 0.5% of the total of my observations). My total sample is a significant part of the total population (50,000 ...
4
votes
0answers
31 views

Predictive model (binary) doesn't seem to fit my own data

I have tried to create a predictive model based on the probit model (common in my field). The model is given as: $$\operatorname{Prob} = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{t}\exp\left(-\frac{x^2}{2}...
4
votes
0answers
118 views

When using L2 regularization outside of linear regression, do the same MAP estimation assumptions hold?

Some context is shared below, and my question is bolded at the end. In the linear regression setting, we learn model weights $\hat{\mathbf{w}}$ to make predictions $\mathbf{\hat{y}}$ from new samples ...
4
votes
0answers
942 views

Understanding the Invariance Property Proof of MLE

There are multiple versions of this question on CV and many proofs are given but I did not fully understand the proof (technique) yet. Theorem. If $$\hat\theta = \operatorname*{argsup}_{\theta\in\...
4
votes
1answer
223 views

What is the likelihood function of this random variable (beta distribution parameterizing a Bernoulli distribution)?

This is related to an earlier self-study question of mine. The setup is that there are $N$ individuals, indexed by $i$, and two time periods. Individuals choose whether to "invent" something in the ...
4
votes
0answers
117 views

Normal Covariance Estimation

I have a hierarchical model and I'm struggling to develop an estimator of the covariance of a normal distribution. This is my specific problem. There are $n$ latent $p$-dimensional vectors, $$\...
4
votes
0answers
51 views

MLE in a stable Gaussian VAR$(p)$ process. Are the Lutkepohl formulas correct?

In «New Introduction to Multiple Time Series», page 90, we have the following formulas for the ML estimators of a stable Gaussian VAR$(p)$ process: where $\tilde \alpha = vec(\tilde A_1,...,\tilde ...
4
votes
0answers
1k views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
4
votes
0answers
653 views

Standard error of the coefficient in GLM

I'm trying to learn about Wald test. I know, that its test statistics is $$ t = \frac{\beta_i}{se\left( \beta_i \right)} $$ But, how is standard error $se$ computed in GLM? I've found only the ...
4
votes
0answers
231 views

when can I substitute an inverse with a pseudo-inverse in an estimator

Short Version: can I substitute the Moore-Penrose generalized inverse of a matrix (R function ginv()) for a matrix inverse (R function ...
4
votes
1answer
988 views

Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$ \theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)}) $$ ...
4
votes
0answers
260 views

Connecting Poisson and multinomial models

Let's say we have multinomial counts $y_{jp}$ (corresponding to observations $j$ over categories $p=1,...P$) that are arranged in a table of $n$ rows and $P$ columns. Then say we have the log-linear ...
4
votes
0answers
259 views

Closed form of Maximum Likelihood Estimator?

I have this Maximum Likelihood (ML) problem, which gives after simplification: $${x_{\hat{\eta}}}^T y_{\hat{\eta}} \times {\mathbb{1}}^T \mathbb{1} - {x_{\hat{\eta}}}^T \mathbb{1} \times {y_{\hat{\eta}...
4
votes
1answer
95 views

Computing the marginal MAP estimate

Suppose I have some IID Gaussian data with priors for mean and standard deviation: \begin{align} P(x|\mu,\sigma)&=\prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}}e^\frac{-(x_i-\mu)^2}{2\sigma^2}\\ p(\...
4
votes
0answers
303 views

Limiting distribution of MLE when true value is on the boundary of the parameter space

We know the nice properties and consistency results of the MLE estimator don't hold when the true value is on the boundary of the parameter space, for eg. if the parameter space is $[0, \inf)$, so ...
4
votes
1answer
629 views

Likelihood of LDA compared to logistic regression

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable: A Linear Discriminant Analysis model where the ...
4
votes
0answers
3k views

What is the exact log-likelihood of an AR(2) model?

Let's say we have the following AR(2) model: $y_t=\phi_0+\phi_1y_{t-1}+\phi_2y_{t-2}+e_t, \; e_t\sim N(0,\sigma^2_e)$ with T observations in total. Working out the conditional log-likelihood is ...
4
votes
0answers
118 views

Instructive figure to explain Maximum Likelihood confidence bands / standard errors

I am trying to better understand and explain maximum likelihood estimation. To explain the intuition of many ML aspects I find it easiest to explain them graphically, like for example the ML-based ...
4
votes
0answers
463 views

Delta Method vs. Lognormal

I have a single parameter $\theta > 0$ of a probability model I estimate with MLE on i.i.d. data. To get rid of the positivity constraint I instead estimate $\log \theta$ for which MLE gives me an ...
4
votes
0answers
2k views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...
4
votes
0answers
275 views

Unbiased Estimator of the truncation points in a truncated normal distribution?

Consider the variables $x_i \text{~} \mathcal{N}(\mu, \sigma^2,a,b)$ iid with truncation points $a$ and $b$, i.e. $a < x_i < b$. Suppose all 4 parameters, namely $\mu, \sigma, a, b$ are all ...
4
votes
2answers
85 views

Is maximum likelihood a form of data substitution? Or not?

I’m using maximum likelihood with missing data. In this case of missing data, is maximum likelihood a form of data substitution? I’m significantly more familiar with multiple imputation which I would ...
4
votes
0answers
89 views

Show that MLE of $\lambda = \frac{n-T_n}{S_n+cT_n}$

$X_i$ are i.i.d exponential, mean $\lambda^{-1}$ for $1 \leq i \leq n$ and, the values are measured such that $X_i = c$ if $X_i \geq c$ and $X_i$ otherwise. Show that MLE of $\lambda = \frac{n-T_n}{...
4
votes
1answer
213 views

Does stable distribution belong to exponential family?

According to Hougaard (1986), positive stable distribution on $\mathbb{R}^+$ belongs to exponential family, how about the case the support of stable distribution being less than zero? The purpose of ...
4
votes
0answers
35 views

Inferences about a distribution given running maximum values

Here is a question inspired by this question from StackOverflow. Suppose you have observations of a variable which is measured once a minute, but the values are only recorded if they are greater than ...
4
votes
0answers
281 views

Maximum likelihood estimation involving both probabilities and probability densities

Note: based on suggestions in the comments, I have rewritten this question. Please refer to the history for the original version. In general my question regards how to compute likelihoods in mixed ...
4
votes
0answers
2k views

Deriving the maximum likelihood for a generative classification model for K classes

In Christopher Bishop's book "Pattern Recognition and Machine learning", there is the following question: Consider a generative classification model for $K$ classes defined by the prior class ...
4
votes
0answers
787 views

Fitting Multivariate Bernoulli distribution

I want to fit a model to a number of observations, each of them being a k-dimensional binary vector $(x_1, x_2, ..., x_k)$ where $x_i \in \{0,1\}$. Naturally I would like to fit a multivariate ...
4
votes
0answers
333 views

What properties of a likelihood function are required for quasi-likelihood estimation?

Quasi-likelihood seems like a great way to use Iteratively Weighted Least Squares to fit linear models with a very general class of likelihoods. But what is that class? Obviously the distribution ...
4
votes
0answers
631 views

2x2 contingency table - maximum likelihood estimate of the odds ratio and exact confidence intervals

I have two questions regarding the following 2x2 contingency table: 1, How can I derive the maximum likelihood estimate of the odds ratio ($OR = (a*d)/(c*b)$) 2, How can I derive exact confidence ...

1
2 3 4 5
18