All Questions
4,023 questions
2
votes
1
answer
64
views
Proof for Asymptotic Normality for MLE with Multiple Parameters
I have a question about how I can prove asymptotic properties of multiple MLE estimators. Most resources you find online give the proof for the case, where only one parameter is estimated (e.g. here: ...
- 315
1
vote
0
answers
71
views
Why the matrix normal distribution is not estimated with this "method of moments"?
According to the Wikipedia Page, a random matrix $\bf{X}\in \mathbb{R}^{p\times q}$ follows a matrix normal distribution $\cal{MN}(\bf M, \bf U, \bf V)$ means that
$$ \text{vec}(\bf X) \sim \cal N ( \...
- 185
4
votes
1
answer
122
views
How is Gaussian log likelihood value calculated in weighted LM, GLM and GLS?
I have examined these R functions glm.fit() gaussian()$aic stats:::logLik.glm() and ...
- 2,373
0
votes
0
answers
41
views
Likelihood ratio test for samples from two different populations
This is basically exactly same as the post here. The answer given over there was not accurate as the null distribution specifically states $\mu_0=\mu_1=0.$
I have attempted it but am not sure if the ...
2
votes
1
answer
90
views
Real-world examples of conditional independence with marginal dependence in cross-sectional data
In a previous question, I explored the derivation of joint likelihood using only conditional independence assumptions - Derivation of Joint Likelihood with Only Conditional Independence Assumptions (...
- 321
3
votes
0
answers
41
views
How do you fit/learn a nonstationary Poisson process from data using maximum likelihood estimation?
I'm trying to understand how to fit/learn a nonstationary Poisson process from data. Through this question, I will explain how I understand the Poisson process, the nonstationary variation of it, and ...
- 131
2
votes
0
answers
38
views
Does MLE on more samples always have a lower risk?
Denote the MLE on $n$ i.i.d. samples with $\hat{\theta}_{n} = \arg\max_\theta \prod_{i=1}^n p(x_i; \theta)$. I wonder if adding more samples always makes the risk lower, or at least not higher. In ...
- 1,817
6
votes
1
answer
226
views
MLE in stochastically increasing parametric family
Let $X$ have cumulative density function $F_{\theta}$, suppose this family is stochastically increasing in $\theta$, that is, for $\theta_1<\theta_2$, $F(x;\theta_2) \le F(x;\theta_1)$.
We have one ...
0
votes
0
answers
22
views
Hypothesis testing with unknown covariance matrix under the alternative hypothesis
I am trying to solve a binary hypothesis testing problem of the form:
$H_1 : \mathbf{y} = \theta \cdot \mathbf{x} + \mathbf{n}$
$H_0 : \mathbf{y} = \mathbf{n}$
where $\theta \in \mathbb{R}$ under $H_1$...
3
votes
1
answer
50
views
Comparisions between REML and ML
I have two questions and really appreciate your answers.
If Maximum Likelihood (ML) and Restricted Maximum Likelihood (REML) methods have the same fixed effects in a model, are the results from these ...
- 31
1
vote
1
answer
71
views
T-test when variance estimator is not sample variance [closed]
It is well known that if $x \sim \mathcal{N}(0, \sigma^2)$ and we have a sample size of $n$ observations of $x$, the distribution of $\frac{x}{\hat{\sigma}}$, where $\hat{\sigma}$ is the sample ...
- 57
11
votes
1
answer
326
views
Transforming Hessian to original parameters after centering and QR-transforming X
I'm starting to more routinely mean-center and QR-rotate design matrices $X$ when doing maximum likelihood estimation (MLE) or Bayesian posterior sampling. This helps with convergence and quickens ...
- 98.5k
3
votes
0
answers
47
views
Is there an analytical solution to the distribution of a sum of observations drawn from a Frechet distribution?
Let $X_i$ be an iid draw from a Frechet distribution. Let $\alpha_i \in \mathbb{R}$.
Is there an analytical expression of the distribution of $\alpha_1X_1 + \alpha_2X_2 + \alpha_3X_3$? That is, can I ...
- 31
2
votes
0
answers
65
views
Doing "maximum p-value estimation" instead of maximum likelihood
Whenever we do maximum likelihood estimation, we look for the parameters that maximize the probability density of the data. On the other hand, when we compute p-values, we look at the tail probability ...
- 475
1
vote
0
answers
33
views
What makes a curve a good fit in the context of logistic regression
As I wanted to gain a better intuition between why separation is a problem in the context of logistic regression, I did create in R two models, one where y is perfectly separated at $x=5$, and one ...
- 81
2
votes
1
answer
154
views
Derivation of Joint Likelihood with Only Conditional Independence Assumptions (Without Marginal Independence)
I'm working through a derivation of the joint likelihood for a dataset $(\mathbf{X}_i, \mathbf{y}_i)$ under the assumption of conditional independence of $\mathbf{y}_i$ given $\mathbf{X}_i$, without ...
- 321
0
votes
0
answers
23
views
Maximum likelihood estimation of binned data
I am currently exploring methods for maximum likelihood estimation with binned data and have observed that many approaches utilize either the multinomial or Poisson distribution to calculate the ...
- 121
1
vote
1
answer
43
views
Unbiased Variance MLE Distribution
If you take $10000$ samples from a normal distribution, the unbiased variance MLE (with Bessel's correction) is
$$\hat{\sigma}^2 = \frac{1}{9999}\sum_i (x_i - \hat{\mu})$$
Apparently the distribution ...
- 503
1
vote
0
answers
17
views
Program for maximum partial marginal likelihood estimator (MPMLE) of regression parameters
I have a system of equations of probabilities which looks like
where . And I would like to solve the above system for U_1, U_2, U_3. I would like to ask is there any package in R or python for these ...
- 195
0
votes
0
answers
22
views
Learning a probability distribution from samples drawn from unknown function
I am wanting to learn some probability distribution $p$ from data (using e.g., Kernel Density Estimation, a Normalizing Flow, whatever your favourite machine learning model is).
If I had a dataset $D =...
- 91
5
votes
2
answers
129
views
What does one do when the Hessian matrix is too sensitive?
I used the survival package in R to calculate the maximum likelihood estimates of Weibull regression co-efficients in R. I then tried to replicate the results by writing the log likelihood function ...
- 63
6
votes
1
answer
245
views
Variance of MLE's in mixture distribution
I am studying mixture models, and I am interested in calculating the variance of the estimators using maximum likelihood. How is the variance calculated in this case? I already implemented the EM ...
- 281
1
vote
0
answers
22
views
Convert units, get different results when fitting extreme value distribution with extRemes
I am using the fevd() and lr.test() functions to examine precipitation using the extRemes R ...
- 11
4
votes
1
answer
236
views
Finding the MLE using a contour plot
When finding the MLE of something using a log-likelihood contour plot, is the MLE in the middle of the middle ring?
- 41
0
votes
0
answers
25
views
How to compare models with different predictors if maximum likelihood estimates of models render false convergence?
I have one full model and several partial GLS models that I want to compare using AIC or BIC to select which factors to include in the final model. I use ML to fit each model before comparing their ...
- 580
1
vote
1
answer
130
views
Wilks' theorem, confidence regions on a projected subspace?
Background
My context is I am thinking about maximum likelihood estimation of a parameter $\theta\in \mathbb{R}^r$ by drawing samples $\{x_i\}$ from a distribution which depends on $\theta$. I know ...
- 146
1
vote
0
answers
19
views
Is there a simple estimate for power transform (Box-Cox, Yeo-Yohnson) transform, that can be computed in O(n)? [closed]
So the power transform defines some function $f(\lambda,y)$ and then we're trying to find the $\lambda$ assuming that $f(\lambda|\mathbf{y}) \sim Normal(\mu,\sigma)$.
This is usually done via MLE ...
- 203
2
votes
1
answer
132
views
Widespread inconsistency in maximum likelihood estimation approach to logistic regression
I have found some widespread inconsistency in how the loss function of logistic regression is derived through the maximum likelihood estimation approach.
In the logistic regression model, we assume ...
- 1,630
1
vote
0
answers
37
views
Max Likelihood of GBM with 2 Markov States
Consider the stochastic process
$$dX_t = \mu_{\epsilon_t}X_tdt + \sigma_{\epsilon_t}X_tdW_t$$
where $W_t$ is a standard Brownian motion. The process $X_t$ is a geometric Brownian motion (GBM) whose ...
- 387
0
votes
0
answers
54
views
Estimate of mean in semiparametric model. Box-Cox fails for negative mean
I have a time series of positive values $X_t \geq 0$ satisfying the following model:
$$\begin{cases}f_*(X_{t}) = f_*(X_{t-1}) + \mu + \varepsilon_{t}, &\forall t \in \{1,2,\dots,T\},\\
X_0 = 1, &...
- 1
0
votes
0
answers
59
views
Terminology for types of errors and uncertainty: intrinsic? fitting?
Say I have a sequence of given variables $x_i$, $i=1,\ldots,n-1$ and the response $y_i$ and we explore the model $y_i\sim \text{Normal}(\alpha x_i,\sigma^2)$ with $y_i$ independent of $y_j$ for $i\neq ...
- 111
2
votes
1
answer
92
views
Optimizing loglikelihood (for Vasicek loss model with autocorrelated systemic risk)
Please find the derivation of this optimization problem below; however, if too long, only the optimization problem is relevant.
Optimization problem
For $T$ observations $x_{1},...,x_{T}\in(0,1)$ and ...
user406757
1
vote
1
answer
85
views
Asymptotic variance of MLE estimator and Fisher Information
I am trying to figure out what I am getting wrong about the following:
Consider an iid Gaussian random variable $X$ with mean $\mu$ and variance $\sigma^2$. Suppose we know $\sigma^2$ and we are ...
1
vote
0
answers
48
views
Hessian for log likelihood of regression with respect to covariance matrix
I am interested in the residual covariance of a multivariate regression model. The regression is
$$
Y_t = X_t \beta + \varepsilon_t
$$
and I have a log likelihood as follows
$$
\mathcal{L}(\Sigma) = \...
- 11
3
votes
2
answers
100
views
Likelihood from forecast::Arima vs. manual replication
I am trying to replicate some results from forecast::Arima in R. I am particularly interested in the likelihood that I would like to use for some likelihood ratio ...
- 69.5k
2
votes
1
answer
128
views
Linear regression MLE of slope; OLS and bivariate MLE
Assume the variables X and Y, sampled from a bivariate normal distribution with mean vector $(\mu, \mu)$,and both having the same population variance $V$, which is known. The only unknown parameter is ...
- 21
1
vote
1
answer
78
views
Expectation of the minimum of random variables (Exponential + Erlang)
Consider the following random variable
$$
Z=\min_i\{X_i+Y_i\}
$$
for $-n\leq i\leq n$, where $X_i\overset{\mathrm{iid}}{\sim}\text{Exp}(\lambda)$, $Y_i\overset{\mathrm{iid}}{\sim}\text{Erlang}(|i|,\...
- 150
3
votes
1
answer
87
views
Maximum of two independent gamma variables
Let $X_1$, $X_2$ be two independent random variables with different gamma distributions, and $X = \max\{X_1, X_2\}$.
Are there known results for the distribution of $X$? Actually I only need to know $\...
- 1,191
1
vote
0
answers
12
views
Analysis of the relative likelihood with parametric bootstrapping
Using parametric bootstrapping, I find that the relative likelihood of model A is $l_A$ and of model B is $l_B$. Repeating the analysis several times yields a distribution of likelihood values for ...
0
votes
0
answers
25
views
Linearity of and pointwise equality in expectation of min() function
Consider the expressions $f = c + s*E[min(a/s, X)]$ and $g = E[min(c + a, c+sX)]$ where
c >= 0
0 < s <= 1
a >= 0
X ~ Poisson($\lambda$/s)
I'd like to think that $f = g$, reasoning as ...
3
votes
1
answer
42
views
Maximum Likelihood Estimation for Pairs of Observations
I have $n$ pairs of observations $(x_i,y_i)$, where each $y_i$ is distributed according to $\text{Pois}(\theta x_i)$, and I wish to do a maximum likelihood estimation for $\theta$ only based on this ...
5
votes
1
answer
611
views
Why is so much of likelihood theory focused on solving the score function equation?
When I first learnt about maximum likelihood estimation, I was told that I should check that the solution of $\frac{d}{d\theta}\text{log}(L(\theta;x_1,...,x_n)) = 0$ was really the maximum likelihood ...
0
votes
1
answer
42
views
Is there an "observation noise" formulation for logistic regression?
In formulating linear regression as the solution to maximum likelihood problem, we need the assumption that the data $X$ and label $Y$ are related by
$$Y = w^TX + \epsilon$$
where $\epsilon$ is a ...
- 671
0
votes
0
answers
16
views
Bishop gradient calculation
In section 3.1.1 of Pattern Recognition and Machine Learning by Christopher Bishop, it is written that
$$\ln p(\mathbf{t} | \mathbf{w}, \beta) = \frac{N}{2} \ln \beta - \frac{N}{2} \ln (2 \pi) - \beta ...
- 11
1
vote
0
answers
16
views
KS test contradicts maximul likely hood fitting
I have two data sets (let call them L and G) that represent a measured physical quantity $\Gamma_n^0(\delta\Gamma_n^0)$. I am performing a Kolmogorov-Smirnov test in ...
- 151
0
votes
0
answers
8
views
Source of the relationship between the number of parameters and data dimensionality
It is commonly understood that, in classical statistical models, the number of observations should be at least equal to the number of parameters in order to ensure the estimability of those parameters,...
- 21
3
votes
0
answers
64
views
Is this a mistake on Wikipedia on Standard Deviation?
On the Wikipedia page about standard deviation, in the section `Estimation', it says
Unlike in the case of estimating the population mean, for which the sample mean is a simple estimator with many ...
- 181
0
votes
0
answers
34
views
How do we derive the standard cross entropy loss from negative log likelihood in a supervised (conditional) learning setting?
I know that when optimizing neural networks (supervised) that cross entropy loss is equivalent to negative log likelihood is euivalent to MLE but I can't get all the math together.
I am trying to ...
- 11
0
votes
0
answers
10
views
Is the significance test in GMM more reliable than it in ML, especially when the regularity conditions are not met?
Maximum likelihood vs generalized method of moments
From the answer to the above question, we can draw a conclusion that GMM and ML can get the same parameters' estimators when the model is exactly ...
- 41
0
votes
0
answers
54
views
Is Arithmetic Mean from the definition of Maximum Likelihood Estimation? [closed]
I am wondering if the arithmetic mean is derived from Maximum Likelihood Estimation or not. While I am studying Maximum Likelihood Estimation for the mean of a binary distribution, the mean was ...
- 3,585