Alright, I expect this is a silly question, but I don't actually know, so.
Suppose there is some random variable that's distributed on the reals, and all I know about the distribution is its mean $\mu$ and its variance $\sigma^2$. Maximum Entropy would have me say that I should choose the Normal distribution as the one that best fit my knowledge about it.
However, suppose I have an utility function about this variable that goes $U(x)=x^4$. Then, my expected utility for drawing from that variable's distribution is $EU(x) = \int\limits_{-\infty}^{+\infty}\!\!x^4\ p(x)\ \mathrm{d}x$. If I choose to use MaxEnt, then that's just $3\sigma^4$. However, if the "true" distribution actually followed by that random variable is, say, the Student's t with $\nu\leq 4$, then my Expected Utility would diverge to infinity.
If I treat the distribution as if I don't know what it is, then I'd have that $p(x)=\int\limits_{D\in\mathcal D}\!\!p(x|D)\ p(D)\ \mathrm{d}D$ where $\mathcal D$ is the space of all possible distributions. In that case, my expected utility is $EU(x) = \int\limits_{-\infty}^{+\infty}\!\int\limits_{D\in\mathcal D}\!\!x^4\ p(x|D)\ p(D)\ \mathrm{d}D\ \mathrm{d}x$ which I expect is not the same as the one I get by using the Normal except for very few priors over $\mathcal D$.
My questions then are: how do I deal with that? What prior distribution over $\mathcal D$ gives me the same result as MaxEnt would? In what sense is the Normal my best guess for my state of ignorance? How do I guarantee that this doesn't diverge? How does having the Normal as the "representative" of my subjective uncertainty take other possible distributions such as the Student's t into account?
Or, rephrasing this: I don't feel like the Gaussian actually represents my state-of-knowledge here, because nowhere in it is it represented that it's possible that the distribution has infinite fourth moment. In what sense is the Gaussian the best description of my state-of-knowledge? Is it, really? If not, what is?
