# Better understanding Maximum Likelihood parameter estimation

Suppose I am trying to model the dependence of a variable $$B$$ on another variable $$A$$ by a function $$B=f(A;k)$$, where $$k$$ is a parameter, whose value I would like to estimate. Given $$n$$ observations $$\{A_i,B_i\}$$, how can use MLE to estimate the value of $$k$$?

For example (and just as an example), I assume the value of glucose ($$=G$$) depends on hemoglobin ($$=H$$) by the following model: $$G=\frac{1}{1+H\cdot k},$$ where $$k$$ is a parameter unique to each individual, that is known to be normally distributed in the population. I am given a set of $$n$$ measurements of glucose and hemoglobin of one person, and would like to estimate the value of $$k$$. How can I do it?

I will be happy to receive recommendations (or, even better, explicit explanations) on resources covering the solution to this problem. I keep finding explanations on estimating a parameter from a distribution, and can't figure out how to relate such explanations to this kind of a problem.

I thought of first isolating $$k$$ to obtain that (EDIT: I failed to correctly isolate $$k$$ in my first thread, and fixed it in an edit) $$k=\frac{1-G_i}{H_i G_i},$$ but wasn't sure how to continue from here...I know that $$k$$ is normally distributed in the population, but how can this help? What if the distribution of $$k$$ in the population wasn't given?

Thank you!

• The core concept of MLE is that it is a probability model that seeks the parameters which maximize the joint probability of observing the data. How would you write down the joint probability of your data according to your model? In what you've written, there doesn't seem to be anything probabilistic, or anything to estimate, because $k$ is a deterministic function of $H$ and $G$, which are known and fixed.
– Sycorax
Aug 18, 2021 at 15:55
• @Sycorax OP did not write explicitly that $H$ and $G$ are "known and fixed," only that they are "measurements". Of course, one would need to assume some kind of measurement noise, but that could be part of an answer, right? I'm a bit confused this question was closed; it seems perfectly answerable to me. Aug 18, 2021 at 16:18
• @jhin The question asks what the MLE would be for some unstated probability model; it is not possible to answer it. If the measurements are, indeed, measured with some amount of error as you have suggested, then that will need to be stated explicitly by OP in an edit, with a description of what the errors are.
– Sycorax
Aug 18, 2021 at 16:34
• @Jorimu In case it's not clear from our discussion until now, your question only makes sense if measurements (of Glucose and Hemoglobin) are noisy, i.e., subject to measurement errors. Otherwise, you could just compute the exact value of $k$ from a single pair of measurements. For an exact answer, you would need to clarify what kind of distribution you're assuming for the measurement noise. Aug 18, 2021 at 17:20
• Unfortunately, I need to reclose the question because it is ambiguous: is your model $G=1/(1+Hk)$ or is it $G = 1-Hk$ as implied later in the post??
– whuber
Aug 18, 2021 at 17:33

In ML estimation, one maximizes the likelihood of the data given the parameters. In your case: $$\hat{k}_{\mathrm{ML}} = \arg\max_k p(G_1, G_2, \ldots, G_N, H_1, H_2, \ldots, H_N \mid k).$$ If we assume that $$G_i = \frac{1}{1+H_i k} + \varepsilon , \quad \varepsilon \sim \mathcal{N}(0, \sigma_\varepsilon ),$$ i.e., we assume Gaussian measurement noise on the $$G_i$$ and independent measurements, we obtain \begin{align} \hat{k}_{\mathrm{ML}} &= \arg\max_k p(G_1, G_2, \ldots, G_N, H_1, H_2, \ldots, H_N \mid k) \\ &= \arg\max_k p(G_1, G_2, \ldots, G_N \mid k) \\ &= \arg\max_k \prod_{i=1}^N p(G_i \mid k) \\ &= \arg\max_k \log \prod_{i=1}^N p(G_i \mid k) \\ &= \arg\max_k \sum_{i=1}^N\log p(G_i \mid k) \end{align} Furthermore, we have that $$p(G_i \mid k) = \mathcal{N}(G_i; \mu_i(k)=\frac{1}{1+H_i k}, \sigma_{\varepsilon })=\frac{1}{\sigma_\varepsilon\sqrt{2\pi}} \mathrm{e}^{-\frac{1}{2}(\frac{G_i-\mu_i}{\sigma_\varepsilon})^2}.$$ Plugging that into the above optimization problem, we get \begin{align} \hat{k}_{\mathrm{ML}} &= \arg\max_k \sum_{i=1}^N -\frac{1}{2}\left(\frac{G_i-\mu_i(k)}{\sigma_\varepsilon}\right)^2 -N\log (\sigma_\varepsilon \sqrt{2\pi}) \\ &= \arg\max_k \sum_{i=1}^N -\frac{1}{2}\left(\frac{G_i-\mu_i(k)}{\sigma_\varepsilon}\right)^2 \\ &= \arg\min_k \sum_{i=1}^N (G_i-\mu_i(k))^2 \end{align}

A few remarks:

• This particular example is an ordinary least squares problem a nonlinear least-squares estimation problem (as can be seen in the last equation).
• If the measurements $$H_i$$ are also assumed to be noisy, things get more complex. In that case, we're dealing with a nonlinear errors-in-variables regression problem. The optimization problem can still be solved numerically, of course.
• What happened to the information that $$k$$ is normally distributed in the general population? That was not used at all in the above derivation, because ML estimation does not consider such priors. If you want to take that into account, you can do maximum a posteriori (MAP) estimation, which is essentially ML estimation + a prior on the parameters. MAP estimation maximizes $$p(k \mid G_1, \ldots, G_N, H_1, \ldots, H_N).$$ Using Bayes theorem, we have that \begin{align} \hat{k}_{\mathrm{MAP}} &= \arg\max_k p(k \mid G_1, \ldots, G_N, H_1, \ldots, H_N) \\ &= \arg\max_k \frac{p(G_{1:N}, H_{1:N} \mid k) \, p(k)}{p(G_{1:N}, H_{1:N})} \\ &= \arg\max_k p(G_{1:N}, H_{1:N} \mid k) \, p(k) \\ &= \arg\max_k \log p(G_{1:N}, H_{1:N} \mid k) + \log p(k) \\ &= \arg\max_k \log p(k) + \sum_{i=1}^N\log p(G_i \mid k), \end{align} where $$p(k)$$ denotes prior knowledge about the distribution of $$k$$. In your example, that could be the population average. We see that this is exactly what we had above for the ML estimate, except for the additional term $$\log p(k)$$.
• I am surprised at the complexity and incompleteness of this answer. It looks like that's because you adopt a model not implied by the question. According to the question, each $(G_i,H_i)$ observation independently determines a $k_i$ assumed to have a Normal$(\mu,\sigma^2)$ distribution. Therefore the MLE of $\mu$ is the arithmetic mean of the $k_i$ and the MLE of $\sigma^2$ is the (non-bias-corrected) variance of the $k_i.$ Although it is indeed more realistic to also model measurement errors, why do you model errors for $G$ but not for $H$??
– whuber
Aug 18, 2021 at 21:07
• @whuber The blockquote at the top of OP says that $k$ has a single value and is a parameter of some function $f$. But your comment supposes a different $k_i$ for each individual, whence we have a distribution over $k_i$. This is why I asked for clarification about the probability model -- it wasn't clear to me what was data and what was a parameter.
– Sycorax
Aug 18, 2021 at 21:46
• @whuber You're completely right that the assumption to model errors for $G$ but not for $H$ was arbitrary. My understanding of the question was that the OP is interested in how to generally derive an ML estimator for a single parameter given a set of i.i.d measurements and a functional model. I showed how to do so using the simpler case of errors in only one variable (but mentioned the EIV case below). The complexity arises because I wanted to show step by step how to get to the final estimator from the ML formulation. Aug 18, 2021 at 22:06
• @whuber Sure, but then it's a MAP estimate, not an ML estimate, isn't it? That's what I was trying to convey in my last remark. (I just spelled out the derivation of the MAP formulation a bit more.) Aug 19, 2021 at 16:16
• Thanks everyone for you suggestions. I think everything is much clearer now. By the way - indeed, for each individual we have a different value of $k$, and I am given a set of several measurements from the same person, from which I would like to estimate $k$. I was missing two key insights - 1) the assumption about the noise. 2) verification that the calculation turns out to be equivalent to OLS. Thanks for accepting my question and answering it... I hope I will manage to better understand everything from here. Aug 20, 2021 at 13:54