I wish to devise a test that determines whether or not an individual is clairvoyant (or if a black-box model works). Let us assume that the clairvoyant believes that they can estimate a person's height (or any other statistic like income whose distribution we know) by their name (or a mental model incorporating multiple factors that we do not know).

We randomly sample $n$ people from the population with heights $h_{i}$, $i \in \{1,2,...n\}$. The clairvoyant gives $n$ intervals of height (in cms) as guesses e.g. $I_{1} = (162, 180), I_{2} = (152, 154)..., I_{n} = (134,155).$ The clairvoyant is deemed correct if a person's height $h_{i} \in I_{i}$. We know what the distribution of height is for the population and we can calculate the probability of a randomly selected person's height falling in an interval. In order to establish whether the individual is a clairvoyant, we need to decide what cut-off we choose for the hit rate (the number of times the clairvoyant in question is correct). How does one compute such a cut-off and how does one devise a test to figure out how competent the clairvoyant in question is? Or is computing errors the only way around this?

  • $\begingroup$ You essentially have a set of Binomial trials, where the probability of success changes with each trial (dependent on what interval the clairvoyant selected). See stats.stackexchange.com/questions/9510/…. You use that to find the distribution of "number of correct guesses" (given the guesses that were made), which you can use to find the likelihood that the person got at least N correct. Of course, you should set what level of evidence you'll require to believe this person is clairvoyant beforehand. $\endgroup$ – Nuclear Wang May 11 at 15:46
  • $\begingroup$ Forecast accuracy metric that involves prediction intervals may be helpful. $\endgroup$ – Stephan Kolassa May 11 at 15:57

Don't use the hit rate as a quality measure for interval predictions. (Or if you do, do not be surprised if your winning algorithm predicts an interval of $(0,300)$ for all instances and gets a hit rate of 100%.)

Your quality measure needs to balance coverage and length of the prediction intervals: yes, we want high coverage, but we also want short intervals.

There is a quality measure that does precisely this and has attractive properties: the interval score. Let $\ell$ and $u$ be the lower and the upper end of the prediction interval. The score is given by

$$ S(\ell,u,h) = (u-\ell)+\frac{2}{\alpha}(\ell-h)1(h<\ell)+\frac{2}{\alpha}(h-u)1(h>u). $$

Here $1$ is the indicator function, and $\alpha$ is the coverage your algorithm is aiming for. (You will need to prespecify this, based on what you plan on doing with the prediction interval. It makes no sense to aim for $\alpha=100\%$ coverage, because the resulting intervals will be too wide to be useful for anything.)

You can then average the interval score over many predictions. The lower the average score, the better. See Gneiting & Raftery (2007, JASA)] for a discussion and pointers to further literature. A scaled version of this score was used, for instance, in assessing predictions intervals in the recent M4 forecasting competition.

Now, as to whether your algorithm is clairvoyant or your black box "works"... well, you will need to figure out whether it is "clairvoyant enough". A clairvoyant should be able to perfectly predict all heights, shouldn't they? So all $u=\ell=h$, and the score should be zero. This sounds like a rather high (or low) bar to clear. So the question really is whether your algorithm is good enough, or better than some competing algorithm or a simply benchmark - for instance, you should certainly test whether your algorithm performs better than just taking empirical intervals over all your training data, which would be the simplest naive benchmark. This may be helpful once you have arrived at this stage.

| cite | improve this answer | |
  • $\begingroup$ This scoring rule would give the same score to a clairvoyant who chooses some interval and consistently sees the true height come in half an interval too low, and one who chooses an interval twice the size and is always correct. I agree that the problem becomes trivial if you allow arbitrarily large prediction intervals, but it seems strange that this might rank an always-wrong prediction as better than an always-right prediction with a bigger interval. $\endgroup$ – Nuclear Wang May 11 at 18:43
  • $\begingroup$ @NuclearWang: I'm afraid I don't quite follow. What is "half an interval too low" and "always correct" in this context? Can you perhaps illustrate with some specific numbers? $\endgroup$ – Stephan Kolassa May 11 at 18:59
  • $\begingroup$ Suppose they predict [140, 160], and the actual height is 130 (half the interval range below the lower bound) - this prediction is incorrect. Taking a=1, the score is 20+2x10 = 40. If the prediction is [130, 170], the score is also 40, but this prediction is correct. Extending this to many samples, the score suggests that the incorrect predictor is just as "good" as the correct one, just because its intervals are smaller. $\endgroup$ – Nuclear Wang May 11 at 19:10
  • $\begingroup$ @NuclearWang: $\alpha=1$ does not make sense, we need a coverage $0<\alpha<1$ for PIs. (Put differently, it doesn't make sense to criticize scoring rules outside their relevant domain of application.) In your first case, $S_\alpha^1=20+\frac{20}{\alpha}$, and in your second case, $S_\alpha^2=40$ independently of $\alpha$, so $S_\alpha^2<S_\alpha^1$ for all $\alpha<1$, as it should be. This is a proper scoring rule, I very much recommend Gneiting & Raftery (2007) on this. $\endgroup$ – Stephan Kolassa May 11 at 19:29
  • $\begingroup$ @NuclearWang: also, I don't think looking at precise outcomes is very helpful. After all, we are evaluating prediction intervals here. So let's posit some true distribution, say $N(0,1)$. Then the interval score is minimized in expectation by the correct $(q_{\frac{\alpha}{2}},q_{1-\frac{\alpha}{2}})$ prediction interval, and there is no possibility for "gaming" it, i.e., getting a lower expected score by specifying some other interval. $\endgroup$ – Stephan Kolassa May 12 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.