2
$\begingroup$

I am looking for the proper statistical terminology to express the fact that one estimation task maybe intrinsically harder to solve than another task. Intuitively, I would characterize this property by the discriminative performance that can be achieved by an optimal classifier trained on infinitely many training data points, and evaluated on an infinitely large hold-out test dataset. In other words: how good can a classifier possibly get on this task?

See, e.g., the following very simple example, in which blue and red are two groups and crosses and circles the two outcomes. The gray line shows the optimal decision threshold for both groups, but model performance will be higher in the blue group compared to the red group.

enter image description here

Does this property of the estimation task have a name? (Notice that I am not interested in a property of a particular estimator (e.g., sharpness) - instead, I am looking for something that characterizes a property of the estimation task.)

Candidates I considered:

  • Aleatoric uncertainty, i.e., one could say that aleatoric uncertainty is higher in one task. This is, however, not what I want to say, because it implies a judgement that it is impossible to get a better separable estimation problem, by, say choosing another measurement.
  • Intrinsic or inherent uncertainty is a term, e.g., in Brier score decompositions. However, this term purely depends on the base rate and is fully independent of how well positive and negative classes can be separated.
  • Separability sounds like a term that means what I want, but the standard definition seems to be as a binary property, i.e., it only captures the extreme case of the classes being fully separable.
  • Input noise, label noise, etc. are possible causes of the property I am interested in.

Is there a technical term for this property of an estimation problem? How would you phrase the fact that one task is "more difficult" than another task?

Improve this question
$\endgroup$
3
  • $\begingroup$ I thought aleatoric uncertainty was exactly this: the random component that cannot be predicted. Why do you not want aleatoric uncertainty as your measure of problem difficulty? $\endgroup$
    – Dave
    Dec 5, 2022 at 14:41
  • $\begingroup$ @Dave At least the definition on wiki emphasizes that aleatoric uncertainty is the uncertainty that is due to true randomness, and not due to something that is currently not measured but in principle measurable: "Just because we cannot measure sufficiently with our currently available measurement devices does not preclude necessarily the existence of such information, which would move this uncertainty into the [epistemic uncertainty] category." $\endgroup$
    – Eike P.
    Dec 5, 2022 at 14:54
  • $\begingroup$ E.g., in my example above, there could be an unmeasured confounder that allows perfect separation of the red group. Then, per the above definition, the difficulty would not be due to aleatoric uncertainty. $\endgroup$
    – Eike P.
    Dec 5, 2022 at 14:56

1 Answer 1

Reset to default
0
$\begingroup$

After some more investigation (prompted by Richard Hardy's addition of the "forecastability" tag, thanks!), I believe one way to phrase this precisely would be that the statistical dependence / mutual information between inputs and outputs is higher in one group compared to the other.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.