I work in a classification problem where collecting instances (labeled or unlabeled) is very expensive. In fact, labeling instances is cheap.
I want to explore an active learning strategy where I do not have access to unlabeled instances, but I can still ask for labeled instances by specifying:
- Statistics on the attributes on the new instances (e.g. give me an instance with an age > 45)
- The instance label (e.g. give me a new instance of label $l$, where $l \in L$ is one of the classification labels) or the label sampling distribution itself.
My questions are:
Does this problem have a name? It looks like a specific case of Active Learning, but I am not sure, since in Active Learning one starts with a set of unlabeled instances, which is not my case.
What types of approaches (from the most rudimentary to the more sophisticated) can I employ to identify the most informative sampling distribution from instance attributes or instance labels?