# Maximum Likelihood estimation with a “dependent parameter”?

Suppose we have a likelihood function

$$L(\theta | X) = f(X; \alpha, \beta, \gamma(\beta))$$

So that the third parameter, $\gamma$, depends on $\beta$ in such a way that they cannot be separated. For example, suppose

$$\gamma (\beta) = \int_0^1 g(s) e^{\beta s} ds$$

for some unknown function $g(s)$. Since we cannot reasonably estimate $g(s)$, we instead wish to replace the integral with a single parameter to estimate. Unfortunately, due to the integral's dependence on $\beta$, we cannot derive maximum likelihood estimates.

I have two questions:

1. Is there any well known procedure for over coming a situation like this?

2. What would happen if we just "ignore" $\gamma$'s dependence on $\beta$? So that we instead consider the function

$$f(X; \alpha, \beta, \gamma)$$

and derive maximum likelihood estimates from that? Would they still be consistent? Can anything "good" be said about this situation?

• If you initially forget all the assumptions about $\gamma$ and simply estimate the three parameters, you will then have estimates $\hat\beta$ and $\hat\gamma$. If you wish you may choose any function $\hat g$ so that $\hat\gamma = \int_0^1\hat{g}(s)e^{{\hat\beta}s}ds$. Thus, you don't seem to have formulated anything any different than a standard probability model and a standard maximum likelihood estimate. This also demonstrates the impossibility of estimating $g$. – whuber Jan 27 '18 at 22:44
• @whuber I guess what I'm asking is, by ignoring the dependence between the two parameters, how would this affect the accuracy of these estimates? Presumably they are no longer MLE estimates, and so none of the nice properties MLE estimates posses are present. So I'm wondering if that route is even worth while – Patty Jan 28 '18 at 6:36
• My point is that the function $g$ is superfluous: it adds nothing to the model and doesn't constrain it in any way. If you make ML estimates of the parameters, they will of course be MLEs. – whuber Jan 28 '18 at 18:54
• I apologise for being slow here, but just to make sure I understand, I'll bring it to a simpler example. Suppose we have an i.i.d sample from a $N(\mu \sigma, \sigma^2)$ distribution. Does this mean that if we let $\kappa = \mu \sigma$ and derived MLE estimates for $\kappa, \sigma$, while ignoring $\kappa$'s dependence on $\sigma$, the resulting estimates would still be MLE estimates and therefore consistent? – Patty Jan 28 '18 at 21:33
• In this case the value of $\sigma$ does not constrain the allowable range of values for the mean $\kappa = \mu \sigma$, so there is no reason to treat $\kappa$ as being dependent on $\sigma$. – Ben Jan 29 '18 at 4:58

To deal with maximum likelihood estimation, you must formulate your problem in such a way that there is a clear range of allowable values for the unknown parameters. In your problem you need to formulate an allowable range of values for the parameter $\gamma$ for each possible value of the dependent parameter $\beta$. To do this you will need to consider what is the allowable range of functions $g$.

For example, suppose you have some independent function $g \in \mathcal{G}$ where the range of this function does not depend on the parameters in your problem. You can then define $\mathcal{Y} (\beta) \equiv \{ \gamma(\beta) | g \in \mathcal{G} \}$ for each allowable value of $\beta$ and you have the partially-maximised likelihood:

$$\bar{L}(\alpha, \beta) \equiv \max_{g \in \mathcal{G}} L(\alpha, \beta, g) = \max_{\gamma \in \mathcal{Y}(\beta)} L(\alpha, \beta, \gamma).$$

This tells you that maximising the likelihood function over the set of allowable functions $g$ is the same as maximising it over the corresponding set of allowable parameters $\gamma$. The latter requires you to specify the range of $\gamma$, and the partially-maximised likelihood then depends on $\beta$ through its direct appearance in the likelihood function, and also through its effect on the allowable range of $\gamma$.

In your problem, you have not been clear about the range of allowable functions $g$ (I am assuming this is some real function). If you impose no constraint on this, you will have $\mathcal{Y} (\beta) = \mathbb{R}$ for all $\beta$, which gives $\bar{L}(\alpha, \beta) = \max_{\gamma \in \mathbb{R}} L(\alpha, \beta, \gamma)$. In this case, there is no problem of parametric dependence and so this is an ordinary maximum-likelihood problem.

Part 1

Can you model $g(s)$ as a polynomial? If so, I believe you could turn $\gamma(\beta)$ into a closed form expression by analytically computing the integral. In general, if you can find some approximation of $g(s)$ that makes the integral analytically solvable, then you're back to ordinary maximum likelihood (without a $\gamma$ at all).

Let's say you don't know much about g(s). But you must know something! So put a Gaussian Process prior on it with mean function F(s). Use a tool like stan, sample the function during an MCMC draw and perform numerical integration within the loop. You can see me trying this here (see line 96). Now it's slow, but it is a way to estimate and incorporate uncertainty in $g(s)$.

If you have a complex nonlinear expression for $g(s)$ but don't want to go the bayesian route, you could incorporate something like the trapezoidal rule in your maximum likelihood routine, but it too will be slow!

Part 2

In order to even speak of dependence between parameters, you have to adopt a bayesian viewpoint, since the classical viewpoint has all parameters as fixed unknown quantities. If you take the classical interpretation, then I think you're okay just estimating $\gamma$ as a constant. In fact, why not use the estimates of $\beta$ and $\gamma$ to gain insight into $g(s)$?