A scoring rule is a means of evaluating an agent's guess of the probabilities associated with a categorical event, given a (categorical) outcome of the event. Depending on the guess and the observed outcome, the scoring rule gives the agent a score (a real number). A scoring rule is supposed to assign scores such that, on average, the agent with the least score makes the most accurate guesses. (Conventions differ as to whether scoring rules are framed in terms of minimization or maximization. Here I'm taking the minimization view.)
An important property of scoring rules is whether they're a proper scoring rule; that is, whether they give the least mean score when an agent guesses the true probabilities (or, in a subjective Bayesian framing, they give the least posterior mean score, given the agent's own priors, when an agent uses its own degrees of belief as its guesses). In the case of a binary event, squared error from 0 or 1 (the Brier score) is a proper scoring rule whereas absolute error is not. Why? Well, the criterion of properness is based on the mean, and the mean is the measure of central tendency that minimizes the sum of squared differences, but need not minimize absolute error.
This line of thinking suggests that if we replace the mean in the definition of a proper scoring rule with some other statistical functional, such as the median, then we'll get an analogous sort of rich family of proper scoring rules. It's not unreasonable to imagine a situation where an agent wants to minimize its median score rather than its mean score. Actually, it seems that there are no nontrivial median-proper scoring rules. Considering the case of a binary event again, if the true probability is less than 1/2, then the median score of an agent will be equal to whatever score is given to the agent when the event doesn't occur, regardless of the event's exact probability. Analogous shenanigans seem to occur if we replace the mean by, say, the geometric mean.
So, is there a sense that in order for the theory of proper scoring rules to work as intended, the statistical functional must be the mean?
I realize this is a vague question, and the best answer is likely to be an explanation of why the question doesn't really make sense, so here's the context where I find myself asking it, to help you un-confuse me. I'm a psychologist of decision-making, and I often find myself wanting to quantify the performance (either predictive performance, under cross-validation, or model fit post-hoc) of a model that spits out probabilities of what people will choose in a binary-decision scenario. The above discussion suggests I should use a proper scoring rule. Annoyingly, proper scoring rules aren't on the same scale as probabilities. I find myself wanting to, for example, take the square root of the mean squared error rather than just looking at the mean squared error (that is, the mean Brier score), but in the case of one trial, the RMSE is equivalent to absolute error, which isn't proper, so wouldn't I then think that models which are less accurate are better? Evidently I can't just change my method of evaluating scoring rules from one in terms of means to one in terms of, e.g., medians. Must I simply familiarize myself with the scale of one of the usual proper scoring rules, or use a signal-detection statistic like area under the ROC curve or d'?
An additional complication is that for one study I'm looking at parametrically bootstrapped model fits, in accordance with Wagenmakers, Ratcliff, Gomez, and Iverson (2004), which means I'm looking at density plots of scores rather than individual scores. Then it's even less clear whether I should be concerned about properness or about some analogous criterion.
Edit: see this comment thread on Reddit for some more discussion.
Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28–50. doi:10.1016/j.jmp.2003.11.004