This is not a direct answer to your question about loss functions, but I am a Statistician, and I use the jargon of my domain, not the jargon of machine learning. I will attempt to answer the question: "which statistic is the best estimator of the population mean?"
It's incorrect to say generally that the arithmetic mean results in a loss of information. In fact in some circumstances, it can be proven that the arithmetic mean or some function of it contains as much information (Fisher information) as the data themselves. This is the concept of a sufficient statistic, i.e. some summary of the data that is sufficient for the data.
For example, if you know that your data follow a Poisson distribution then the sufficient statistic is $T(X) = X_1 + ... + X_n$. Which is simply the sum of the data. For a Normal distribution where you know the variance then the arithmetic sample mean is the sufficient statistic for the population mean. That is, it contains all of the information and no other statistic will do better. Now granted, we are never in the situation where we know are data are normally distributed and happen to know exactly the variance. But that is why we have the central limit theorem. Even for skewed data, if what you really care about is the population mean, then the arithmetic mean is pretty good bet, especially if you have a lot of observations. So to that end, I would say in a lot of circumstances, especially when you have a lot of observations the arithmetic mean is best if what you care about is the population mean.
Now, if you happen to be in the privileged position of knowing your data come from some other distribution, perhaps some pathological negative exponential distribution, then you're correct there may be a better sufficient statistic. In that circumstance the sufficient statistic for $\mu$ is the minimum observation. This is favorite example of Mukhopadhyay in Probability and Statistical Inference and you will find all exercises you can stomach in there to demonstrate.
To answer your question more generally, about how to choose the best statistic: plot your data. Look at it. Think about where it came from and how it was collected. Think about what it is you are actually trying to make inference on, and whether the way these data were collected is actually appropriate for that. Think about the form your data take: Are they strictly integer data? Proportions with a known denominator? Are they skewed, if so would a log-normal make a for a good approximation? Choose a parametric family that seems to satisfy and caveat if you must.