I have a model as follows:

$y_i = \theta x_i + \eta_i, i=1,2,...,N$

where $y_i$ and $x_i$ are known observations greater than $0$, the $\eta_i\sim Gamma(a,b), a>1$. Now, I want to obtain the model parameter $\theta$ using the maximum likelihood estimation. First, the likelihood function is given as:

$L(\theta ) = {\left( {\frac{{{b^a}}}{{\Gamma (a)}}} \right)^n}.\prod\limits_{i = 1}^N {({y_i} - \theta {x_i})^{\alpha-1}} .\exp [ - b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )]$

then logarithmic

$\log L(\theta ) \propto (\alpha-1)\log \left( {\prod\limits_{i = 1}^N {({y_i} - \theta {x_i})} } \right) - b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )]$

the first derivative of $\theta$ is

$\frac{{\partial \log(L)}}{{\partial \theta }} = (\alpha-1) \sum\limits_{i = 1}^N {(\frac{{ - {x_i}}}{{{y_i} - \theta {x_i}}}} ) + b\sum\limits_{i = 1}^N {{x_i}} $

Obviously, estimation of $\theta$ cannot be directly obtained from the equation $\frac{{\partial L}}{{\partial \theta }}=0$ due to the complex expression.

To solve this problem, the folloing methods are considered:

(1) Can $\sum\limits_{i = 1}^N {(\frac{{ - {x_i}}}{{{y_i} - \theta {x_i}}}} )$ be accurately approximated as a simple expression.

(2) Can translate the problem of $\max(\log L(\theta))$ into a simple optimization problem without using differential method.

How to get the estimation of $\theta$ ? Hope to give some suggestions, thank you very much


1 Answer 1


The likelihood is missing the term $\alpha-1$, because the gamma PDF contains $x^{\alpha-1}$: $$-\log L(\theta ) \propto -(\alpha-1)\sum\limits_{i = 1}^N {\log {({y_i} - \theta {x_i})} } + b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )$$

Minimizing this expression is a convex optimization problem with the constraints $y_i-\theta x_i>0$. The second part is affine, and the first part is affine composition of a convex function ($-\log x$). Sum of two convex functions is also convex. So, this problem can be solved easily via a CVX solver. Or you can even apply gradient descent yourself.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.