Questions tagged [maximum-likelihood]

a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

984 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
13
votes
0answers
1k views

Testing for a significant difference between ML estimates: Likelihood ratio or Wald test?

I am trying to test whether or not there is a significant difference between maximum likelihood estimates of two genetic parameters (selection and dominance) across two environments with genotype data ...
10
votes
0answers
566 views

Does using bootstrapped samples improve parameter estimates for a fitted distribution?

The R package retimes has a function for fitting an ex-Gaussian distribution to a set of observations. The method involves taking multiple bootstrapped samples of the observations, and fitting the ex-...
7
votes
0answers
306 views

When using L2 regularization outside of linear regression, do the same MAP estimation assumptions hold?

Some context is shared below, and my question is bolded at the end. In the linear regression setting, we learn model weights $\hat{\mathbf{w}}$ to make predictions $\mathbf{\hat{y}}$ from new samples ...
7
votes
0answers
306 views

when can I substitute an inverse with a pseudo-inverse in an estimator

Short Version: can I substitute the Moore-Penrose generalized inverse of a matrix (R function ginv()) for a matrix inverse (R function ...
7
votes
0answers
258 views

asymptotic unbiasedness of weibull mle

It's known that the MLEs of the two-parameter Weibull distribution scale and shape parameters are not available in a closed form. It is, however, known that they do exist, are unique, and moreover, ...
6
votes
0answers
201 views

Why does MLE tend to normal distribution

We have $X_1,\dots, X_n$ are iid (the distribution can be of any type, e.g. Bernoulli (p), normal ($\mu, \sigma^2$), Poisson ($\lambda$). If we use MLE $\hat \theta$ to estimate any parameter $\theta$ ...
6
votes
0answers
171 views

Maximum likelihood and degenerate Fisher information

I am wondering if there are some standard results to find rates of convergence of the MLE for different sub-parameters, when the Fisher information is degenerate. More precisely, suppose that I ...
6
votes
0answers
139 views

Bayesian inference via approximate data likelihood

Suppose that we have a very large i.i.d. sample $x_1,...,x_n$ and a data likelihood defined by $$p(x | \theta,\beta) = \prod_ip(x_i | \theta,\beta)$$. Further suppose that $\theta$ is the parameter ...
6
votes
0answers
1k views

Why is Fisher Scoring easier to compute?

In practice, the observed information matrix (Newton-Raphson) is usually replaced by its expectation, known as Fisher scoring. Link: https://en.wikipedia.org/wiki/Scoring_algorithm#Fisher_scoring ...
6
votes
0answers
739 views

Likelihood maximization: MCEM algorithm versus MCMC algorithm

Hello Everyone this is my first question. I am a particle physicist and I am doing some empirical studiues on parameters estimation using different methods (this might give me some handle to study on ...
6
votes
0answers
2k views

Deriving the maximum likelihood for a generative classification model for K classes

In Christopher Bishop's book "Pattern Recognition and Machine learning", there is the following question: Consider a generative classification model for $K$ classes defined by the prior class ...
6
votes
0answers
2k views

"weight" input in glm.nb function in R. How exactly does the weight affect the likelihood?

I would like to understand how the weight argument of glm.nb is affecting the likelihood function. I understand that glm.nb find ...
6
votes
0answers
158 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
6
votes
0answers
2k views

Parameter Estimation for Naive Bayes - Maximum a posteriori and Maximum Likelihood

I am wondering if I understand those terms correctly. To summarize my thoughts: In naive Bayes, our decision rule is basically the Maximum a posteriori (MAP) estimate of our hypothesis. We assign an ...
6
votes
0answers
596 views

Intuitive explanation of "integrate out random effect"

We are trying to figure out an intuitive reasoning behind integrate out the unobserved random effect. The specific formula is: $f\big(y_i|x_i;\beta, \sigma_c\big)=\int_{-\infty}^{+\infty}\Big(\prod_{...
6
votes
0answers
3k views

Frequency weights, rare events and logistic regression

I'm working on a model that requires me to look for predictors for a rare event (less than 0.5% of the total of my observations). My total sample is a significant part of the total population (50,000 ...
6
votes
0answers
712 views

Computing Standard Errors in EM algorithm

I'm applying the EM to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov chain, ...
5
votes
0answers
86 views

Maximum likelihood estimate for multivariate sum of normal distributions

For each $j = 1,\dots,N$, let $\mu_j \in \mathbb{R}^N$ denote a known column vector, $\Sigma_j \in \mathbb{R}^{N\times N}$ a known covariance matrix, and $\theta_j \in \mathbb{R}$ an unknown parameter,...
5
votes
0answers
84 views

Does Fisher scoring always outperform Newton optimization?

My understanding is that Fisher scoring has several advantages over Newton raphson optimization such as Computational efficiency: if certain conditions are met (example:During MLE estimation, if link ...
5
votes
0answers
107 views

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
5
votes
0answers
112 views

What are some of the robustness checks for the likelihood ratio test?

In the application of statistical methods in social science, one usually does a lot of robustness checks. If I got some publishable findings using LRT test by discrimination two theoretical models, ...
5
votes
0answers
321 views

Nonnegative Matrix Factorization as Maximum Likelihood

Elements of Statistical Learning has this on such NMF loss function (section 14.6 Non-negative Matrix Factorization): The matrices $\mathbf{W}$ and $\mathbf{H}$ are found by maximizing $$ L(\mathbf{...
5
votes
0answers
2k views

Odds ratio MLE calculation: unconditional vs conditional?

I am running an odds ratio calculation for site methylation amongst cases and controls. In this situation is it preferable to use a conditional or unconditional MLE? I am asking because R uses a ...
5
votes
0answers
737 views

Standard error of the coefficient in GLM

I'm trying to learn about Wald test. I know, that its test statistics is $$ t = \frac{\beta_i}{se\left( \beta_i \right)} $$ But, how is standard error $se$ computed in GLM? I've found only the ...
5
votes
1answer
1k views

Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$ \theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)}) $$ ...
5
votes
0answers
352 views

Relationship of Poisson regression to determining proportions in a mixture distribution

I am a particle physicist, and a very frequent task is: given data sampled from a distribution mixture $$ Z \sim \sum_i \phi_i F_i, $$ where $\phi_i$ are the mixture proportion / prior probabilities ...
5
votes
0answers
329 views

Unbiased Estimator of the truncation points in a truncated normal distribution?

Consider the variables $x_i \sim \mathcal{N}(\mu, \sigma^2,a,b)$ iid with truncation points $a$ and $b$, i.e. $a < x_i < b$. Suppose all four parameters, namely $\mu, \sigma, a, b$ are unknown. ...
5
votes
0answers
339 views

Bayesian estimates for Deming regression coinciding with least-squares estimates

Consider the following Deming model with independent replicates : $$x_{i,j} \mid \theta_{i} \sim {\cal N}(\theta_{i}, \gamma_X^2), \quad y_{i,j} \mid \theta_{i} \sim {\cal N}(\alpha+\beta\theta_{i}, \...
5
votes
0answers
46 views

Inferences about a distribution given running maximum values

Here is a question inspired by this question from StackOverflow. Suppose you have observations of a variable which is measured once a minute, but the values are only recorded if they are greater than ...
5
votes
0answers
939 views

Fitting Multivariate Bernoulli distribution

I want to fit a model to a number of observations, each of them being a k-dimensional binary vector $(x_1, x_2, ..., x_k)$ where $x_i \in \{0,1\}$. Naturally I would like to fit a multivariate ...
5
votes
1answer
13k views

Conditional vs. Unconditional Maximum Likelihood

I have some questions on the difference between conditional MLE (CMLE) and unconditional MLE (UMLE) in practice. In what follows I will only talk about the unconditional and conditional mean and leave ...
5
votes
0answers
340 views

Estimating right-censored data

I am VERY new to stats. I have a large amount of life-time data (delay in arrival since start of experiment) from repeat experiments. Some data is missing, but essentially represents a delay longer ...
5
votes
0answers
4k views

Gamma distribution and Cramér-Rao bound

There are two definitions of the GAMMA distribution: I did the ML estimation, generated the Fisher Information, compared it to the Variance and the Cramer Lower Bound was reached, so the estimator ...
4
votes
1answer
707 views

Parameter uncertainity in least squares optimization: rescaling Hessian

Given a least squares optimization problem of the form: $$ C(\lambda) = \sum_i ||y_i - f(x_i, \lambda)||^2$$ I have found in multiple questions/answers (e.g. here) that an estimate for the covariance ...
4
votes
0answers
51 views

This mixed-type family of random variables has no dominating measure, so a likelihood function can't be defined?

Let $$ X = \begin{cases}\theta & \text{with probability 1/2}\\ Z\sim N(0,1) & \text{with probability 1/2.} \end{cases}$$ Here, $\theta\in\mathbb{R}$ is the parameter to be estimated. It ...
4
votes
0answers
32 views

Predictive model (binary) doesn't seem to fit my own data

I have tried to create a predictive model based on the probit model (common in my field). The model is given as: $$\operatorname{Prob} = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{t}\exp\left(-\frac{x^2}{2}...
4
votes
0answers
548 views

James Stein Estimator for more than one Sample

I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
4
votes
0answers
1k views

Understanding the Invariance Property Proof of MLE

There are multiple versions of this question on CV and many proofs are given but I did not fully understand the proof (technique) yet. Theorem. If $$\hat\theta = \operatorname*{argsup}_{\theta\in\...
4
votes
0answers
251 views

GP: How to select a model for a classification task, based in overall accuracy and log-marginal likelihood?

I have fitted a Gaussian Process (GP) to perform a binary classification task. The dataset is balanced, so I have an equal number of samples with 0/1 label for the training. The covariance function ...
4
votes
1answer
398 views

What is the likelihood function of this random variable (beta distribution parameterizing a Bernoulli distribution)?

This is related to an earlier self-study question of mine. The setup is that there are $N$ individuals, indexed by $i$, and two time periods. Individuals choose whether to "invent" something in the ...
4
votes
0answers
119 views

Normal Covariance Estimation

I have a hierarchical model and I'm struggling to develop an estimator of the covariance of a normal distribution. This is my specific problem. There are $n$ latent $p$-dimensional vectors, $$\...
4
votes
0answers
194 views

Estimating variance of MLE estimate of Beta-Geometric/NBD without MCMC

I'm using Fader's BG/NBD model for customer LTV calculations. The log likelihood is the following: $$ \sum_{i=1}^{N} \ln L(r, \alpha, a, b|X_i=x_i, t_{x_i}, T_i) = \frac{B(a, b + x)}{B(a, b)}\frac{\...
4
votes
0answers
53 views

MLE in a stable Gaussian VAR$(p)$ process. Are the Lutkepohl formulas correct?

In «New Introduction to Multiple Time Series», page 90, we have the following formulas for the ML estimators of a stable Gaussian VAR$(p)$ process: where $\tilde \alpha = vec(\tilde A_1,...,\tilde ...
4
votes
0answers
1k views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
4
votes
0answers
72 views

Automatic fitting of normalization constant as a parameter in noise contrastive estimation

In the paper on Noise Contrastive Estimation, the authors define a parameterized density function $p_m^0\left(x;\alpha\right)$ to estimate the unnormalized PDF of the data, and then further define a ...
4
votes
0answers
742 views

Implementing ARMA Log Likelihood with the Kalman Filter Algorithm

A popular algorithm to determine the (complex) log likelihood function of an ARMA(p,q) process involves generating it through the use of a state-space model and the Kalman Filter. I started reading ...
4
votes
0answers
299 views

Connecting Poisson and multinomial models

Let's say we have multinomial counts $y_{jp}$ (corresponding to observations $j$ over categories $p=1,...P$) that are arranged in a table of $n$ rows and $P$ columns. Then say we have the log-linear ...
4
votes
0answers
303 views

Closed form of Maximum Likelihood Estimator?

I have this Maximum Likelihood (ML) problem, which gives after simplification: $${x_{\hat{\eta}}}^T y_{\hat{\eta}} \times {\mathbb{1}}^T \mathbb{1} - {x_{\hat{\eta}}}^T \mathbb{1} \times {y_{\hat{\eta}...
4
votes
1answer
129 views

Computing the marginal MAP estimate

Suppose I have some IID Gaussian data with priors for mean and standard deviation: \begin{align} P(x|\mu,\sigma)&=\prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}}e^\frac{-(x_i-\mu)^2}{2\sigma^2}\\ p(\...
4
votes
1answer
752 views

Likelihood of LDA compared to logistic regression

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable: A Linear Discriminant Analysis model where the ...

1
2 3 4 5
20