3
$\begingroup$

One justification for the ubiquity of the (multivariate) normal distribution in statistical/machine learning modeling is that it maximizes entropy among distributions with mean $\mu$ and variance matrix $\Sigma$ (see for example, here). More generally, any exponential family maximizes entropy among distributions subject to the parameters governing the moment restrictions related to its sufficient statistics. The link above suggests that this maximum entropy property is useful because it makes "minimal assumptions" about the true underlying distribution. But in what sense is this true? It seems like depending on your task, for example, using a normal distribution is far from making "minimal assumptions". For example, if we are worried about outliers, using a normality assumption masks the true uncertainty. For what ends exactly is using a maximum entropy parameterization a good way to make "minimal" assumptions? Or is maximum entropy an end in itself?

$\endgroup$
0