What is the minimal amount of parameters that can be retained in a particular context?

  • 2
    $\begingroup$ What exactly do you mean by "retained" in this case? $\endgroup$ – gung - Reinstate Monica Oct 5 '19 at 11:52
  • $\begingroup$ For five independent normal distributions, you would need to have the population means and standard deviations of the five distributions, and the percentage of the mixture for each normal distribution. I guess that makes 15, and they're all essential. $\endgroup$ – BruceET Oct 5 '19 at 19:12
  • $\begingroup$ @Bruce The five proportions are related by the sum-to-unity equation, leaving only 14 parameters. $\endgroup$ – whuber Oct 5 '19 at 20:08
  • $\begingroup$ @whuber: Right about the 14 proportions, thanks. $\endgroup$ – BruceET Oct 5 '19 at 20:26

If the dimension is $n$, you need $n$ parameters for each mean vector and $(n^2+n)/2$ parameters for each covariance matrix in general. Additionally, if the number of mixture components is $m$, $m-1$ parameters are needed for mixture composition weights since they need to sum up to $1$. Special constraints over covariance matrices reduce this number of parameters needed.

| cite | improve this answer | |
  • $\begingroup$ are the $m-1$ parameters you mention the mixture weights themselves, or are they other supporting parameters? wouldn't there have to be $m$ mixture weights if there are $m$ mixture components because each weight is assigned to a different component? $\endgroup$ – develarist Jul 13 at 13:45
  • $\begingroup$ @develarist yes, they're mixture weights. $m-1$ is enough because they'll add up to $1$, and the last one can be found from others. $\endgroup$ – gunes Jul 13 at 13:47
  • $\begingroup$ but the expectation-maximization algorithm will in fact actively solve for all $m$ mixture weights, right $\endgroup$ – develarist Jul 13 at 13:49
  • 1
    $\begingroup$ The algorithm should be decoupled from the model. E-M is not the only way of solving GMMs. In general, if you can derive a parameter from others, it's not counted towards free parameters. $\endgroup$ – gunes Jul 13 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.