Techniques for strategically crafting a ML dataset For a supervised machine learning application where the input features can be readily calculated and the corresponding labels are the result of a somewhat time-consuming simulation using the inputs, it is advantageous to "craft" the best dataset with the fewest data. We have several ideas to this end, a few are:

*

*Use unsupervised learning techniques to build an initial set of inputs that maximally spans the feature space.

*After an iteration of training, generate new inputs and add to the training set if the trained model is uncertain of the input class (predicted probability near 0.5 for a binary system).

*Add new inputs to the training set based on whether they are similar to previous inputs that were misclassified (using unsupervised learning)

I am wondering if any of these are valid approaches and more broadly if there is a name for techniques likes this where a ML dataset is strategically crafted and improved in an iterative manner?
This is different than most data augmentation techniques I'm familiar with because rather than processing existing inputs (e.g. rotating, translating images) in this application it's trivial to generate entirely new inputs.
 A: As the other answer suggests, this is, firstly, a very broad question and, secondly, an open research question.
The strategy you're proposing (start from a small set, then iteratively actively select new datapoints based on some criterion) is known as active learning; hundreds of methods have been proposed for that. The uncertainty-based sampling strategy you describe is one common approach. Here's the depressing conclusion, AFAIK: in general, very few methods perform consistently better than just sampling new datapoints at random. This just seems to be a very hard problem.
Other strategies for limiting the need for labeled data include

*

*Semi-supervised learning: use some labeled and typically a lot more unlabelled data, which you mentioned, and for which many strategies have been proposed such as pseudo-labeling / automatic labeling. Here's a review on SSL.

*Weakly supervised learning: for example, you might want to train an
image segmentation model but only have image-level labels, which are
much cheaper to obtain. Here's an example paper.

*Use synthetic data, either from a first-principles simulation model or as some kind of recombination of real data points. Doing this in such a way that the model really learns something that generalizes to reality isn't trivial, though. Here's a book on the subject. The simplest version of this that almost everybody in image processing does nowadays is, of course, training data augmentation: rotating, stretching, deforming, blurring, cropping, ...

Returning to the general question of how to craft a useful dataset: I think a lot of evidence now points to the fact that you will want to maximize diversity in every possible dimension that will be relevant in the real-world application: demographics, recording modalities and institutions, noise levels and types, and whichever other dimensions the thing you're measuring can vary along. If you can simulate your datapoints, that might mean trying to span as much of the simulation model's parameter space as is feasible with your selected sample size budget.
A: This is clearly an open and active research question. To my knowledge it is relevant in a few different contexts:
1/ In a supervised learning setting, it can be extremely tedious (for humans) to label the training data. In that case it is preferable to label the samples that "matter the most".
2/ In Scientific Machine Learning, ground truth data is usually generated by computationally expensive ODE or PDE solvers, which limits the number of available samples.
3/ In sensor-based ML, such sensors (cameras, satellites, thermometers, etc...) can be expensive/difficult to monitor, which limit the number of available sensors.
You may take a look at a few recent papers:

*

*Increasing Data Diversity with Iterative Sampling to Improve Performance

*Global field reconstruction from sparse sensors with Voronoi tessellation-assisted deep learning

*Gaussian processes for autonomous data acquisition at large-scale synchrotron and neutron facilities

*Exploring Representativeness and Informativeness for Active Learning

To my knowledge, the problem of "crafting" small but representative datasets is very popular in the domain of Active Learning as well as data-centric AI.
