Can a classifier be used to improve its own training data?

Question

Introduction:

Consider a classification problem of $\mathbb{R}^n$ into $\mathbb{R}^2$. Let $\mathcal{U}$ be a set of instances whose class is unknown, but can be discovered paying a cost $\gamma$ for each instance.

Goal:

To create a Machine-Learning classifier capable of predicting the class of a new instance from $\mathcal{U}$ with an $\alpha$% accuracy (AUC, or any other metric), minimizing the total cost, which is given by $\gamma$ times the number of instances for which the class has to be discovered.

Solution:

The solution is to subselect instances from $\mathcal{U}$, discover their classes and collect them into a training dataset $\mathcal{T}$.

Problem:

The cost is $C = \gamma \times |\mathcal{T}|$. How to optimize $C$?

Consider the following scenarios:

1. instances are randomly selected from $\mathcal{U}$, discovered, and collected into a training dataset. The process stops when the predictor achieves a threshold goodness.

2. instances are randomly selected from $\mathcal{U}$, the predictor is executed on them obtaining the predicted classes. Then, instances of the more present class in $\mathcal{T}$ are ruled out, while only the others's class is discovered, paying $\beta$ for each one of them. This way, the presence of both classes in $\mathcal{T}$ will be evened by the own predictor, which will in theory lead to a better performance.

Questions:

What do you think about scenario 2)? Is it reasonable? Am I missing something? I've been searching on the internet but didn't find anything similar to my problem. Do you know any similar study?

Answer 1

• Usually it is a bad idea to reinvent the wheel, so if you need this to solve an applied problem, it would be wise to find already existing solution. It's not exactly the same, but you may be interested in learning more about concepts such as active learning, or human-in-the-loop in machine learning if you consider non-automated solutions as well.
• The problem of actively sampling the data is often treated as multi-armed bandit problem in discrete case, or Bayesian optimization (Frazier, 2018) in general, with methods such as Gaussian processes for continuous cases. TL;DR those algorithms learn the distribution of the data and can help you to make a decision what kind of data is most promising to sample. Such approaches were used in clinical trials (Giovagnoli, 2021, Takahashi and Suzuki, 2010 for some reviews) and other scenarios.
• If I understand correctly, in your second scenario you want to discard the data. Why would you do that? Throwing out valid data is almost never a good idea, especially if it is expensive to obtain. If you're sampling randomly this should not be a problem. Having unbalanced data is often not a problem. If it is, you can always use an algorithm less susceptible to it, use survey weights to adjust for it, etc.