How to choose loss function (in unbounded parameter space)?

Question

How does one choose a loss function for a given problem? (I've looked through stackexchange, and I haven't been able to find a thread that discusses this.)

Let say I observe some data $x \in \mathbb{R}^n$, and I'm interested in estimating some parameter $\theta$ related to the distribution from which $x$ came. Suppose that $\theta$ lies in an unbounded parameter space. Suppose that I'm interested in an estimator of the form: $\hat\theta = \arg\min_\theta \mathcal{L}(x, \theta)$, where $\mathcal{L}$ is some loss function. Is there a principled way to choose $\mathcal{L}$?

Some comments:

• One way to choose the loss function would be through a probabilistic modeling approach. For instance, using squared error in regression could be motivated as being naturally derived from the likelihood of a gaussian with spherical covariance. However, this loss function has received some justification in this context beyond normality assumptions from the Gauss-Markov theorem.
• The loss functions I've seen for GLMs are also derived from probabilistic considerations. However, we know that the estimated coefficient is robust in that it will still be consistent even if the distribution is misspecified (as long as the link is correctly specified and all of true covariates are included.) (See Gourieroux, C., A. Monfort, and A. Trognon. 1984. Pseudo maximum likelihood methods: theory. Econometrica 52: 681–700.) This provides some (asymptotic) robustness to the choice of loss function.
• When doing nonnegative matrix factorization, you can consider the negative likelihood when assuming that the matrix entries are drawn from a Poisson distribution. However, we can show that the loss function derived from this is equivalent to (generalized) kullback-leibler divergence. See, for instance, Relationship between Poisson generation and generalized Kullback-Leibler divergence. This is nice since it provides an information-theoretic motivation for the loss function.
• The loss function could be chosen by practical considerations such as computational efficiency. Surely squared error has received such mileage for this reason.
• Perhaps, if the situation calls for such a property, the loss function can be derived in an ad-hoc way by introducing asymmetry into a symmetric loss function.
• In machine learning, objective functions are often used that are not derived from probabilistic modeling; however, I'm not sure if those objective functions should be considered loss functions.

Is there some theory concerning the choice and effect of loss function?

I'm also interested in the choice of loss function when determining admissibility of a given estimator. So, in this setting, an estimator is just given, and we're interested in determining whether the risk (or expected loss) of the estimator is ever dominated by another estimator. Is there a framework for choosing a loss function in this setting that's different in the setting where we use the loss function to produce an estimator?

Answer 1

Canonically your loss function should reflect the losses you sustain from forecasting error. For instance, when your weather forecast tells you there's no rain today, while in fact it rains, you call a taxi to get home instead of walking and it costs you \$10. Alternatively, the forecast projected rain, so you drove to work instead of walking, which cost you \$2 of gas. So, your loss function is (false rain: 1, false clear: 5)

In a real world people don't bother to construct the loss functions of their problems, but use one of the default choices such as RMSE or MAE. Why? Mainly, because they don't know better, but also because it's "difficult".

Note, that the loss function is not related to probabilities or likelihoods at all. You minimize the probability weighted loss, i.e. each loss will be linked to the probability, then the expected loss is minimized.