4
$\begingroup$

I need to understand how a particular statistical challenge has been formally recognised or is commonly described in literature, and what the best academic resources are that discuss it. Here's the problem: consider you need to detect a rare phenomenon and in classifying the whole population (no need to sample) to detect it you want to consider the effects of different hypothetical rates of classification error. To illustrate assume the following hypothetical reality:

  • total population (N) = 1 million
  • rare phenonon of interest is present in 1 in 1,000 of the population (P) and absent in the remaining 999,000 (A)

A 100% accurate classification would therefore yield 1,000 true positives and 999,000 true negatives. But we cannot assume our classification is perfect. But let's assume our false negatives are relatively few (certainly having a negligible effect on the estimate for A). The real problem is the effect that even a tiny rate of false positive classification will have on estimation of P, since if just 1 in 1,000 of A observations are misclassified the estimate for P is doubled.

Grateful for any insight community can offer for how this has been formalised and where I can find more information. It feels like the domain of Bayesian statistics which I'm not very expert in.

Improve this question
$\endgroup$
12
  • $\begingroup$ You need to start by asking if the misclassification costs are unequal, i.e. are false-positive errors more costly than false-negative ones, and if so by how much. It may be that assigning all patterns to the majority class is the optimal solution in terms of minimising the expected cost, in which case just do that, but it also may allow meaningful classification. See stats.stackexchange.com/questions/539638/… . However, even in that case you can still estimate tendencies as @FrankHarrell suggests. $\endgroup$
    – Dikran Marsupial
    Commented Nov 11, 2023 at 23:03
  • 1
    $\begingroup$ The body of the question seems inconsistent with the title. Do you just want to estimate the prevalence of the phenomena in the population or do you want to identify the cases of that phenomena in the population? $\endgroup$
    – Dikran Marsupial
    Commented Nov 11, 2023 at 23:06
  • $\begingroup$ I should clarify the context here is establishing confidence in existing classifications in our data based on the way it has been collected. Let's say we are dealing with self-reported survey data. $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:13
  • 1
    $\begingroup$ Exactly, we only have the data. But it seems to me we should still be able to make some assessment about reliability of the minority subgroup's population estimate based on plausible hypothetical rates of misclassification and given the ostensible size difference between majority and minority subgroups that the data is telling us. It seems to me this has to be a well established problem in statistics, which is what I'm trying to pin down. $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:37
  • 1
    $\begingroup$ I'm not sure how that would establish confidence in the classification as it would just be looking at the effect of your confidence in the classifier ("plausible hypothetical rates") on the population probability. Of course you could estimate them using a simulation study (estimate the true numbers of positive and negatives from the assumed misclassifcation rates) but it wouldn't be meaningfully establishing confidence AFAICS $\endgroup$
    – Dikran Marsupial
    Commented Nov 11, 2023 at 23:44

2 Answers 2

Reset to default
5
$\begingroup$

If you just want to estimate the prevalence of the phenomenon in the population, rather than identify the the particular instances of the phenomenon, then I would use e.g. logistic regression to make a probabilistic classifier that estimates the probability of class membership and then apply the algorithm of Saerens et al. which can be used to adjust the output of a probabilistic classifier to compensate for unknown operational class frequencies that are not the same as those in the training data (it is an EM-type algorithm). As a by product it gives you an estimate of the operational class frequencies.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Dikran. As per my other comments we're talking about evaluating confidence in existing data classifications rather than designing a classifier. $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:18
  • $\begingroup$ @geotheory I'm sorry but it is unclear what you mean by "evaluating confidence in existing classifications". $\endgroup$
    – Dikran Marsupial
    Commented Nov 11, 2023 at 23:22
  • $\begingroup$ My comment in main thread should help to clarify $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:23
5
$\begingroup$

This topic has been covered to the extreme on this site. Briefly, when either class A or class B are extremely rare, classification is an inappropriate goal because as you have so rightly noted you can just classify everyone as being in the majority category and it would be hard for any classifier to beat the accuracy of doing that. The inappropriateness of classification is discussed in detail here. Instead the goal is to estimate tendencies, i.e., probabilities, using a direct probability model such as binary logistic regression. Probabilities also form the basis for optimum decision making and for withholding action if probabilities are in the gray zone.

This is not a Bayesian vs. frequentist issue.

Improve this answer
$\endgroup$
3
  • $\begingroup$ The hbiostat.org link is dead is there an updated URL? $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:07
  • $\begingroup$ I think I need to clarify that the context here is not statistical modelling and I'm not talking about performance of a classifier per se but rather the existing classifications in our data. This is about the confidence we can have in our data based on the way it has been collected. Let's say we are dealing with self-reported survey data. $\endgroup$
    – geotheory
    Commented Nov 11, 2023 at 23:10
  • $\begingroup$ Sorry - link fixed now. $\endgroup$
    – Frank Harrell
    Commented Nov 12, 2023 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.