Refers to a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample. Maximum likelihood estimators are consistent, are efficient (in that they achieve the Cramer-Rao lower bound) and are asymptotically normal ...
17
votes
1answer
748 views
Intuition behind why Stein's paradox only applies in dimensions $\ge 3$
Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff ...
14
votes
3answers
420 views
What are some illustrative applications of empirical likelihood?
I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012).
In my efforts to understand it, I have gleaned ...
14
votes
2answers
2k views
What is the difference between a partial likelihood, profile likelihood and marginal likelihood?
I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
13
votes
4answers
723 views
How to ensure properties of covariance matrix when fitting multivariate normal model using maximum likelihood?
Suppose I have the following model
$$y_i=f(x_i,\theta)+\varepsilon_i$$
where $y_i\in \mathbb{R}^K$ , $x_i$ is a vector of explanatory variables, $\theta$ is the parameters of non-linear function ...
12
votes
2answers
295 views
Which distributions have closed-form solutions for maximum likelihood estimation?
Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?
11
votes
5answers
934 views
Advanced statistics books recommendation
There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
11
votes
4answers
5k views
Using lmer for prediction
Hello
I have two problems that sound like natural candidates for multilevel/mixed models, which I have never used. The simpler, and one that I hope to try as an introduction, is as follows:
The data ...
11
votes
3answers
467 views
How to do estimation, when only summary statistics are available?
This is in part motivated by the following question and the discussion following it.
Suppose the iid sample is observed, $X_i\sim F(x,\theta)$. The goal is to estimate $\theta$. But original sample ...
11
votes
2answers
379 views
Hessian of profile likelihood used for standard error estimation
This question is motivated by this one. I looked up two sources and this is what I found.
A. van der Vaart, Assymptotic Statistics:
It is rarely possible to compute a profile likelihood ...
10
votes
3answers
485 views
Can the empirical Hessian of an M-estimator be indefinite?
Jeffrey Wooldridge in his Econometric Analysis of Cross Section and Panel Data (page 357) says that the empirical Hessian "is not guaranteed to be positive definite, or even positive semidefinite, for ...
10
votes
1answer
473 views
Maximum likelihood estimators for a truncated distribution
Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
10
votes
1answer
423 views
What are the disadvantages of the profile likelihood?
Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter.
If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed ...
9
votes
2answers
371 views
The trinity of tests in maximum likelihood: what to do when faced with contradicting conclusions?
The Wald, Likelihood Ratio and Lagrange Multiplier tests in the context of maximum likelihood estimation are asymptotically equivalent. However, for small samples, they tend to diverge quite a bit, ...
9
votes
1answer
242 views
Hypothesis testing on the inverse covariance matrix
Suppose I observe i.i.d. $x_i \sim \mathcal{N}\left(\mu,\Sigma\right)$, and wish to test $H_0: A\ $vech$\left(\Sigma^{-1}\right) = a$ for a conformable matrix $A$ and vector $a$. Is there known work ...
8
votes
2answers
2k views
When should I *not* use R's nlm function for MLE?
I've run across a couple guides suggesting that I use R's nlm for maximum likelihood estimation. But none of them (including R's documentation) gives much theoretical guidance for when to use or not ...
8
votes
3answers
463 views
Is there always a maximizer for any MLE problem?
I wonder if there is always a maximizer for any maximum (log-)likelihood estimation problem? In other words, is there some distribution and some of its parameters, for which the MLE problem does not ...
8
votes
1answer
353 views
Properties of logistic regressions
We're working with some logistic regressions and we have realized that the average estimated probability always equals the proportion of ones in the sample; that is, the average of fitted values ...
8
votes
4answers
708 views
How large should a sample be for a given estimation technique and parameters?
Is there a rule-of thumb or even any way at all to tell how large a sample should be in order to estimate a model with a given number of parameters?
So, for example, if I want to estimate a ...
7
votes
5answers
760 views
What is the 'fundamental' idea of machine learning for estimating parameters?
The 'fundamental' idea of statistics for estimating parameters is maximum likelihood. I am wondering what is the corresponding idea in machine learning.
Qn 1. Would it be fair to say that the ...
7
votes
2answers
1k views
How do I know which method of parameter estimation to choose?
There are quite a few methods for parameter estimation out there. MLE, UMVUE, MoM, decision-theoretic, and others all seem like they have a fairly logical case for why they are useful for parameter ...
7
votes
2answers
293 views
How can I estimate 95% confidence intervals using profiling for parameters estimated by maximising a log-likelihood function using optim in R?
How can I estimate 95% confidence intervals using profiling for parameters estimated by maximising a log-likelihood function using optim in R?
I know I can asymptotically estimate the covariance ...
7
votes
1answer
115 views
MLE/Likelihood of lognormally distributed interval
I have a variable set of responses that are expressed as an interval such as the sample below.
...
7
votes
1answer
239 views
Observed information matrix is a consistent estimator of the expected information matrix?
I am trying to prove that the observed information matrix evaluated at the weakly consistent maximum likelihood estimator (MLE), is a weakly consistent estimator of the expected information matrix. ...
7
votes
1answer
226 views
Distribution of reciprocal of regression coefficient
Suppose that we have a linear model $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ that meets all the standard regression (Gauss-Markov) assumptions. We are interested in $\theta = 1/\beta_1$.
Question ...
6
votes
5answers
531 views
If you use a point estimate that maximizes $P(x | \theta)$, what does that say about your philosophy? (frequentist or Bayesian or something else?)
If somebody said
"That method uses the MLE the point estimate for the parameter which maximizes $\mathrm{P}(x|\theta)$, therefore it is frequentist; and further it is not Bayesian."
would you ...
6
votes
2answers
206 views
How much calculus is necessary to understand maximum likelihood estimation?
I am trying to plan out a study plan for learning MLE. In order to do this I am trying to figure out what is the minimum level of calculus that is necessary to understand MLE.
Is it sufficient to ...
6
votes
2answers
113 views
For what distribution is a trimmed mean the maximum likelihood estimator?
The sample mean is the maximum likelihood estimator of $\mu$ for a normal distribution $\text{Normal}(\mu,\sigma)$. The sample median is the maximum likelihood estimator of $m$ for a Laplace ...
6
votes
1answer
139 views
Multinomial choice with binary observations
Is there a standard name for a multinomial choice model where the observations are in the form of binary questions such as "do you prefer A to B" and "do you prefer B to D"? This seems like a common ...
6
votes
2answers
68 views
How to incorporate costs (into logit model) of false positive, false negative, true positive, true negative if they are different costs?
How to incorporate costs (into logit model) of false positive, false negative, true positive, true negative responses, if they are different costs ? Is it possible to do that on the level of ...
6
votes
3answers
248 views
Finding the MLE for a univariate exponential Hawkes process
The univariate exponential Hawkes process is a self-exciting point process with an event arrival rate of:
$ \lambda(t) = \mu + \sum\limits_{t_i<t}{\alpha e^{-\beta(t-t_i)}}$
where $ t_1,..t_n $ ...
6
votes
2answers
289 views
A good book with equal stress on theory and math
I have had enough courses on statistics during my school years and at the university. I have a fair understanding of the concepts, such as, CI, p-values, interpreting statistical significance, ...
6
votes
1answer
208 views
Minimum-Distance estimation of mixed/mixture distributions
Please note: I posted this first on Mathoverflow. Someone there advised me that on stats.stackexchange the question might fit better here. This is the link to the original post.
I currently have to ...
6
votes
2answers
358 views
Full information maximum likelihood for missing data in R
Context: Hierarchical regression with some missing data.
Question: How do I use full information maximum likelihood (FIML) estimation to address missing data in R? Is there a package you would ...
5
votes
2answers
2k views
Estimating Lambda for Box Cox transformation for ANOVA
Assumptions:
In an ANOVA where the normality assumptions are violated, the Box-Cox transformation can be applied to the response variable. The lambda can be ...
5
votes
3answers
506 views
How can I work around “lumpiness” in simulated maximum likelihood estimation?
I am attempting to estimate a model of the following form:
W = alphaH * H + alphaM * M + alphaL * L + X * beta
where H, M, L ...
5
votes
1answer
175 views
What is the UMVUE for $\sigma^2$ in $\mathcal N(0, \sigma^2 )$?
By using the exponential class factorization theorem, I came up with $Y = \sum (x_i)^2$ to be the complete and sufficient statistics for $\sigma^2$ .
Using this sufficient statistic as a condition, ...
5
votes
1answer
145 views
Linear regression with shot noise
I'm looking for the right statistical terminology to describe the following problem.
I want to characterize an electronics device that has a linear response
$Y = \beta_0 + \beta_1 X + \epsilon$
...
5
votes
1answer
94 views
Estimation by future likelihood maximization
Background
Conventional approaches to fitting a priori models to observed data seek to find those model parameters that maximize the likelihood of the data. For more complicated models, this ...
5
votes
0answers
84 views
Bayesian, MDL or ML interpretation of cross-validation?
Is there any known Bayesian, ML or MDL interpretation of cross-validation? Can I interpret cross validation as performing the right update on a specifically crafted prior?
5
votes
0answers
254 views
Testing for a significant difference between ML estimates: Likelihood ratio or Wald test?
I am trying to test whether or not there is a significant difference between maximum likelihood estimates of two genetic parameters (selection and dominance) across two environments with genotype data ...
4
votes
3answers
435 views
Does MLE require i.i.d. data? Or just independent parameters?
Estimating parameters using maximum likelihood estimation (MLE) involves evaluating the likelihood function, which maps the probability of the sample (X) occurring to values (x) on the parameter space ...
4
votes
2answers
266 views
How to bootstrap the best fit distribution to a sample?
If I have a sample:
set.seed(0)
x <- rlnorm(500)
Then I can use the fit.distr function to find the best fit among two candidate distributions, e.g.
...
4
votes
2answers
131 views
Determinant perturbation approximation
The following problem comes from a max likelihood calculation for gaussian families, but is of independent interest.
Is it possible to find a closed-form approximation for small values of $x$ for
...
4
votes
3answers
765 views
Recreating traditional null hypothesis testing with Bayesian methods
I am trying to recreate (in R) a frequentist hypothesis testing in Bayesian from, by calculating Bayes factors of the null (H0) and alternative (H1) models.
The model is simply a simple linear ...
4
votes
3answers
725 views
What is numerical overflow?
I came across an error of numerical overflow when running a maximum likelihood estimation on a log-linear specification.
What does numerical overflow mean?
4
votes
1answer
720 views
Calculating likelihood from RMSE
I have a model for predicting a trajectory (x as a function of time) with several parameters. At the moment, I calculate the root mean square error (RMSE) between the predicted trajectory and the ...
4
votes
1answer
114 views
What are speed differences beetwen ML implementations in different languages?
I am trying to write my own ML library. For speed reasons I started out writing things in C using BLAS, but then I learned that NumPy and Theano also use BLAS. I am wondering if there are huge speed ...
4
votes
3answers
542 views
Optimization of MLE for mixture problems
I have about 1000 data points from some thick tailed distribution that I would like to fit a parametrized distribution to. From my data, I've made some adjustments and constructed an empirical ...
4
votes
2answers
840 views
Maximum likelihood estimation in a Poisson model for football (soccer) scores
I've got a set of football results and I want to make a probabilty model of football scores as described in Dixon, Coles (1997, http://www.math.ku.dk/~rolf/teaching/thesis/DixonColes.pdf). They ...
4
votes
2answers
163 views
List of likelihood-based classification techniques
This is a basic statistical pattern recognition question.
I'm aware of LDA classification, Naive Bayes Classification techniques which give output as a likelihood (of data belonging to a certain ...
