I would like to fit multiple distributions that share one of their parameters. As a simple example, let's say I have two different datasets which I know follow a Gaussian distribution. let's say I know $\mu_1$ and $\mu_2$, and now I want to estimate $\sigma_1$ and $\sigma_2$. However, I know that $\sigma_1=\sigma_2$.

Now, I would like to find this joint parameter $\sigma$ that best fits both datasets. The maximum likelihood approach tells us how to find parameters by maximizing the likelihood of one distribution, but how do I maximize the likelihood of both? Perhaps I should maximize the sum of the likelihoods or the product of the likelihoods?

  • $\begingroup$ In your simple normal distribution example, the MLE of $\sigma^2$ is simply $(n_1+n_2)^{-1}(\sum_{i=1}^{n_1}(x_i-\bar{x}_{n_1})^2+\sum_{i=1}^{n_2}(y_i-\bar{y}_{n_2})^2)$. Here $x_1,\ldots,x_{n_1}$ and $y_1,\ldots,y_{n_2}$ are two samples with different means and identical variance. $\endgroup$ – semibruin Jul 22 '13 at 3:46
  • $\begingroup$ @semibruin thanks, I gave that as an example but I am more interested in a case where the MLE does not have a closed form. $\endgroup$ – Bitwise Jul 22 '13 at 11:50

In general, for independent samples, maximize the product of both likelihoods, or equivalently the sum of the log-likelihoods, over all parameters. Consider the samples as one data-set with joint probability density $$f_{\vec{X}}(\vec{x})=f(x_{11};\mu_1,\sigma)\cdot f(x_{12};\mu_1,\sigma) \cdot \ldots \cdot f(x_{1n_1};\mu_1,\sigma)\cdot f(x_{21};\mu_2,\sigma)\cdot f(x_{22};\mu_2,\sigma) \cdot \ldots \cdot f(x_{2n_2};\mu_2,\sigma)$$ It's no different from what you do in a regression model where each covariate pattern gives its own expected value for the response. As @semibruin points out, there's a simple analytic solution for the case of two normal distributions with common variance but different means.

  • $\begingroup$ Great, that's what looked most natural to me. Thanks! $\endgroup$ – Bitwise Jul 22 '13 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.