a method of estimating parameters of a statistical model by choosing the parameter value that optimizes the probability of observing the given sample.

learn more… | top users | synonyms (1)

2
votes
1answer
35 views

Motivation for gradient descent method over canonical method (for OLS/MLE) for simple linear regression?

I am beginner in machine learning and I am currently trying to find the motivation for gradient descent method. I am confused why we want to employ gradient descent method for linear regression? I see ...
1
vote
0answers
17 views

Bayesian estimates for Deming regression coinciding with least-squares estimates

Consider the following Deming model with independent replicates : $$x_{i,j} \mid \theta_{i} \sim {\cal N}(\theta_{i}, \gamma_X^2), \quad y_{i,j} \mid \theta_{i} \sim {\cal N}(\alpha+\beta\theta_{i}, ...
0
votes
0answers
19 views

Box-Cox transformation for the ordered outcome model

I wonder if there is someone out there who had the following problem. Namely, I am trying to fit an ordered logit model (-ologit-) in Stata but before that I would ...
0
votes
0answers
14 views

Information matrix at Wald statistic

AFAIK, the Wald statistic of ML estimator uses the limit normality of MLE, and it looks as: assume to test $H_0 : \theta = \theta_0$ $T = (\hat{\theta} - \theta_0)' (I(\theta_0))^{-1} (\hat{\theta} ...
1
vote
0answers
44 views

Fitting GLM with Quasi-Newton method

I'm trying to code my own quasi-Newton algorithm for fitting GLMs in R. My results do not match up with glm and I've been over my code many, many times so I'm ...
0
votes
0answers
23 views
+50

Finding quality of fit for two discrete variables with low statistics

I have data from an experiment which I am trying to explain using a model. I do not have an analytic formula for the prediction of the model but instead I got its prediction through a simulation. The ...
0
votes
1answer
14 views

log multivariate normal differentiation (MLE)

I've come across a lot of explanations of how to differentiate the multivariate normal, but they all appear to skip the step that I'm stuck on. Here's what I've got so far. By logging and removing ...
1
vote
0answers
29 views

Convergence from the EM Algorithm with bivariate mixture distribution

I have a mixture model which I want to find the maximum likelihood estimator of given a set of data $x$ and a set of partially observed data $z$. I have implemented both the E-step (calculating the ...
2
votes
0answers
43 views

Maximizing combined log likelihood from different dataset using r

I have observed data $y$. And I have a function that gives me estimated $\hat{y} = f(x,\hat{P})$ where P is the parameter I want to estimate. I was able to optim() command in R to get maximized ...
2
votes
1answer
22 views

On FIML assumptions

In Hayashi's Econometrics, page 529, he states one of the assumptions we need for the FIML estimator. My doubt is in the third line of point 1). He says that the vector ...
0
votes
1answer
40 views

Weibull distribution

I need to find a distribution that fail regularity condition.Maybe weibull distribution can be but I did not find why, please help me.
0
votes
0answers
12 views

Practical application of Cramer-Rao lower bound to calculate the variance of estimator

I would like to use the Cramer-Rao lower bound to help me estimate the variance of my maximum likelihood estimator, across a range of different samples of data. My question is, how do I do this ...
3
votes
2answers
99 views

What to do when your likelihood function has a double product with small values near zero - log transform doesn't work?

I currently have a likelihood function defined as the following: $$ L=\prod_{i=1}^{N}\left[\prod_{s=1}^{S_i}L_{is}(y\space|\space \rho_A)\times\phi + \prod_{s=1}^{S_i}L_{is}(y\space|\space ...
0
votes
0answers
56 views

How can the R-matrix in a mixed model be estimated?

In Henderson's Mixed Model equation: $y = X\beta + Zv + \epsilon$ where the joint variance of v and the error term is: $Var\begin{bmatrix} v \\ \epsilon \end{bmatrix} = \begin{bmatrix} G & ...
0
votes
0answers
11 views

How to estimate Mean and Variance of a Gaussian dataset of 20 numbers using ML,MAP and Bayesian Inference?

I have generated 20 random numbers from a Gaussian distribution with mean 5, and standard deviation 1. I have a question that has asked me to estimate the mean and variance using the above methods. ...
1
vote
0answers
19 views

On the computational efficiency of some matlab code when finding the loglikelyhood

I am looking at some code on GAS models found here (number 7). The code finds the log-likelihood of the joint conditional distribution of each observation by summing up the single log-likelihood of ...
3
votes
1answer
56 views

Connection between MLE (Maximum Likelihood Estimation) and introductory Inferential Statistics?

The first thing that one learns in statistics is to use the sample mean, $\hat{X}$, as an unbiased estimate of the population mean, $\mu$; and pretty much the same would be true for the variance, ...
0
votes
1answer
34 views

How to estimate parameters of a log-normal distribution?

I am using income data from the Current Population Survey for a small undergrad economics paper. In economics, there is evidence that the income of 97%–99% of the population is distributed ...
2
votes
1answer
118 views

Conceptual question on log-likelihood value

I am trying to implement the log-likelihood expression Eq(7) from the paper, Parameter Estimation for Linear Dynamical Systems (1996). Re-writing, For the model, $h(t) = \mathbf{A^T} h(t-1) + ...
5
votes
1answer
75 views

Do I get the nice asymptotic properties of MLE when I restrict the parameter space?

I would like to know if the MLE is still consistent, asymptotically normal, and efficient when I put restrictions on the parameter space. I think my confusion stems from the definition of the ...
2
votes
0answers
34 views

MLE estimate of normal distribution

I am quoting this from Greene's econometrics book: The occasional statement that the properties of the MLE are only optimal in large samples is not true, however. It can be shown that when ...
1
vote
1answer
37 views

How to combine heckman selection and binary endogenous variable in a two-step way?

I want to fit a probit model with a binary endogenous variable and heckman sample selection problem, it's something like ...
0
votes
0answers
20 views

MLE of sum of two binomials

Let's $X_1 \sim Binomial(n_1, p_1), X_2 \sim Binomial(n_2, p_2)$. I want to find MLE of $\psi = p_1 - p_2$. There is a property of MLE: if $\hat{\theta}$ is MLE of $\theta$, then $g(\hat{\theta})$ is ...
4
votes
0answers
20 views

Can I penalize an arbitrary regression model and get Elastic-Net-esque results?

Consider an arbitrary-ish regression model with the unpenalized likelihood $$ \log \mathcal{L} = \sum_i f\left(y_i\,|\,g(\beta_0 + \beta x_i)\right) $$ with $\beta = \left(\beta_1, \dots, ...
2
votes
1answer
37 views

For Maximum-likelihood estimation (MLE), must one assume a probability distribution of the dependent value, the error term, or both?

So to estimate the parameters of a model using MLE one must write the likelihood function of having observed the data sample at hand by assuming that it came from a particular distribution. In order ...
0
votes
0answers
12 views

Fitting gamma distribution to data set with one zero observation

I am using maximum likelihood estimation to fit a gamma distribution to shelf life data. Specifically, the data I have is the time (in days) between the day a product was sold and the day the first ...
4
votes
2answers
375 views

I am wondering why we use negative (log) likelihood sometimes?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
0
votes
0answers
8 views

Model Selection In small clusters

I have a question please. Is ok to make model selection with MLE in small cluster in order to allow for comparison, and after getting the final model then fit the final model with REML? Since REML is ...
2
votes
0answers
20 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
3
votes
1answer
53 views

Self Study: ML Parameter Estimates — do I need numerical maximization?

I have a particular PDF with two parameters, specified as: $$\alpha \beta e^{-\beta x}(1 - e^{-\beta x})^{\alpha - 1}, \alpha > 0, \beta > 0, x_i > 0$$ Given a random iid sample $(x_1, ...
1
vote
0answers
13 views

Fit a conditional Bernoulli-gamma distribution using maximum likelihood

I am trying to model payment amounts for a collections agency. I am struggling with formulating the function I need to optimize, however. I have researched various distributions and stumbled-across ...
0
votes
0answers
16 views

Maximum likelihood estimation of the log-normal distribution using R [migrated]

I'm trying to estimate a linear model with a log-normal distributed error term. I already have working code for a linear model with normally distributed errors: ...
0
votes
1answer
31 views

Problem with generalized likelihood ratio test from samples from beta distribution

I was trying to resolve this exercise: This exercise is from the book "Statistical Inference, Second Edition" by Casella and Berger. Checking the solutions manual, I was understanding the solution ...
1
vote
1answer
38 views

What are the (philosophical) assumptions behind GMM and Maximum Likelihood Estimation?

As stated in the question. In particular, how does a researcher know when to apply which estimation method and are there any examples that can show when one case is more appropriate than the other? ...
2
votes
0answers
32 views

Finding MLE with ordered statistics?

Let Y1 < Y2 < ... < Yn be the order statistics of a random sample of size n from the uniform distribution of the continuous type over the closed interval: $$[\theta - \rho, \theta + \rho]$$ ...
1
vote
0answers
18 views

Is the application of the Frisch-Waugh-Lovell Theorem really necessary?

Suppose I have a model \begin{eqnarray} y = X_1 \beta + X_2 \gamma + \epsilon \\ X = Z \Pi + V \end{eqnarray} where $X_1$ is endogenous, Z are instruments, $X_2$ are exogenous. If I however include ...
1
vote
1answer
13 views

Finding covariance matrix for MLE of correlated outputs

I generate data using the following model: $\begin{pmatrix}Y_1\\Y_2\end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix}\mathbf{X}\beta_1\\\mathbf{X}\beta_2\end{pmatrix}, \mathbf{\Sigma} \right)$ I ...
1
vote
1answer
19 views

How to find $\arg\max$ of a neural network?

Let's say I have a neural network $f$ that takes input $\vec x \in \mathbb {R}^n$ and produces output $f(\vec x) \in \mathbb{R}$. How can I find $\hat x = \underset{\vec x}{\arg\max} \; f(\vec x)$?
1
vote
0answers
27 views

Deming Regression/Errors-in-Variables with replicates

I have a question about a model related to Deming regression and would appreciate some help and/or publications to further study this model. Statistical Model: \begin{align} ...
0
votes
2answers
40 views

Computationally efficient Gaussian MAP estimation algorithm in MATLAB

I have a MAP estimation model for a Gaussian prior and i.i.d Gaussian noise: $$y=x+n$$ where $x\sim\mathcal{N}(0,\Sigma)$ and $n\sim \mathcal{N}(0,\sigma^2I)$. The MAP estimate is given by $$ ...
4
votes
1answer
38 views

Distributed Datasets and MLE

Suppose I have a very large dataset of size N, evenly distributed over M computers so that each computer has N/M data points. Suppose I want to fit a model using MLE that requires an iterative method. ...
1
vote
1answer
42 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) $f(x,y) = ...
0
votes
1answer
39 views

Is LIML consistent under heteroskedastic errors?

Please let the answer be yes. Suppose we have a model \begin{eqnarray} y= X \beta + \epsilon \\ X = Z \Pi + V \end{eqnarray} and we compute the LIML estimator \begin{eqnarray} \hat{\beta}_{LIML} = ...
0
votes
0answers
11 views

Is this R code of Rao score test for the Bernoulli data model correct? [migrated]

I am a complete statistical noob and new to R, hence the question. I've tried to find an implementation of the Rao score for the particular case when one's data is ...
0
votes
0answers
21 views

Penalized ML estimation of non-linear probit

I have a model of the form $P(y_i=1) = \Phi(\frac{w_1^{\beta'x_i}-w_2^{\beta'x_i}}{\sigma' x_i})$ where $y_i$ is a binary response, $\Phi$ is the normal CDF, $w_1$ and $w_2$ are non-negative ...
4
votes
1answer
32 views

Negative weights in maximum likelihood method?

In physics we like to use the maximum likelihood method to fit our models to our data. (I'm sure the first part of this post is review to you all, I just want to be complete so that you will know ...
0
votes
1answer
75 views

Optimizing parameter estimates by minimizing chi^2 in iterative procedure

I need to minimize my Chi^2 (bottom-left in figure 1) by adjusting parameter-values in a MLE-procedure (or something alike). The chi^2 (red) is a goodness-of-fit measure. It expresses how well the ...
0
votes
0answers
16 views

GARCH estimation, reduce outlier effect with t distribution

I am trying to estimate GARCH(1, 1) using MLE. As my data contains a lot of outliers, I think using t distribution to represent errors makes a lot of sense. But I am completely lost at how to ...
0
votes
1answer
25 views

Deriving the common LIML estimator from first principles

David Hendry (1976) comments that deriving the LIML estimator is hard. I tend to agree. Guido Imbens has a nice expression here which reads \begin{eqnarray} \hat{\beta}_{LIML} = (X'(I - \lambda M_Z) ...
4
votes
2answers
59 views

Expectation maxmisation algorithm increases true likelihood at each iteration

I've heard that the EM algorithm ensures that the true likelihood is non-decreasing at each iteration of the algorithm, but I'm not sure why this is the case. I've provided a basic plot which I ...