I am looking for the proper statistical terminology to express the fact that one estimation task maybe intrinsically harder to solve than another task. Intuitively, I would characterize this property by the discriminative performance that can be achieved by an optimal classifier trained on infinitely many training data points, and evaluated on an infinitely large hold-out test dataset. In other words: how good can a classifier possibly get on this task?
See, e.g., the following very simple example, in which blue and red are two groups and crosses and circles the two outcomes. The gray line shows the optimal decision threshold for both groups, but model performance will be higher in the blue group compared to the red group.
Does this property of the estimation task have a name? (Notice that I am not interested in a property of a particular estimator (e.g., sharpness) - instead, I am looking for something that characterizes a property of the estimation task.)
Candidates I considered:
- Aleatoric uncertainty, i.e., one could say that aleatoric uncertainty is higher in one task. This is, however, not what I want to say, because it implies a judgement that it is impossible to get a better separable estimation problem, by, say choosing another measurement.
- Intrinsic or inherent uncertainty is a term, e.g., in Brier score decompositions. However, this term purely depends on the base rate and is fully independent of how well positive and negative classes can be separated.
- Separability sounds like a term that means what I want, but the standard definition seems to be as a binary property, i.e., it only captures the extreme case of the classes being fully separable.
- Input noise, label noise, etc. are possible causes of the property I am interested in.
Is there a technical term for this property of an estimation problem? How would you phrase the fact that one task is "more difficult" than another task?