Parameter estimation under Gamma noise distribution

Question

I have a model as follows:

$y_i = \theta x_i + \eta_i, i=1,2,...,N$

where $y_i$ and $x_i$ are known observations greater than $0$, the $\eta_i\sim Gamma(a,b), a>1$. Now, I want to obtain the model parameter $\theta$ using the maximum likelihood estimation. First, the likelihood function is given as:

$L(\theta ) = {\left( {\frac{{{b^a}}}{{\Gamma (a)}}} \right)^n}.\prod\limits_{i = 1}^N {({y_i} - \theta {x_i})^{\alpha-1}} .\exp [ - b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )]$

then logarithmic

$\log L(\theta ) \propto (\alpha-1)\log \left( {\prod\limits_{i = 1}^N {({y_i} - \theta {x_i})} } \right) - b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )]$

the first derivative of $\theta$ is

$\frac{{\partial \log(L)}}{{\partial \theta }} = (\alpha-1) \sum\limits_{i = 1}^N {(\frac{{ - {x_i}}}{{{y_i} - \theta {x_i}}}} ) + b\sum\limits_{i = 1}^N {{x_i}} $

Obviously, estimation of $\theta$ cannot be directly obtained from the equation $\frac{{\partial L}}{{\partial \theta }}=0$ due to the complex expression.

To solve this problem, the folloing methods are considered:

(1) Can $\sum\limits_{i = 1}^N {(\frac{{ - {x_i}}}{{{y_i} - \theta {x_i}}}} )$ be accurately approximated as a simple expression.

(2) Can translate the problem of $\max(\log L(\theta))$ into a simple optimization problem without using differential method.

How to get the estimation of $\theta$ ? Hope to give some suggestions, thank you very much

Answer 1

The likelihood is missing the term $\alpha-1$, because the gamma PDF contains $x^{\alpha-1}$: $$-\log L(\theta ) \propto -(\alpha-1)\sum\limits_{i = 1}^N {\log {({y_i} - \theta {x_i})} } + b\sum\limits_{i = 1}^N {({y_i} - \theta {x_i}} )$$

Minimizing this expression is a convex optimization problem with the constraints $y_i-\theta x_i>0$. The second part is affine, and the first part is affine composition of a convex function ($-\log x$). Sum of two convex functions is also convex. So, this problem can be solved easily via a CVX solver. Or you can even apply gradient descent yourself.