Calculating Potential Usefulness of Acquiring Additional Data Imagine Anne has a labeled training dataset for a machine learning prediction problem. There is an opportunity to acquire more data from an agent, at a cost. However, before she decides to acquire that data by paying the cost, she wants to know if that additional data is likely to improve her model or not.
You can assume that there exists a black-box mechanism that allows Anne to perform some low cost computations on that additional data or the combined data (to explore the usefulness of that data). But she can NOT train a new machine learning model using the new data before she pays the non-refundable cost.
What kind of computations Anne should consider to get an idea/intuition of the added value this new data may bring? For example, if she could calculate a few metrics on the additional data or on the combined data, what should those metrics be?
How would your answer change if this was an unsupervised machine learning problem (e.g. clustering), and the datasets were unlabelled.
A few examples: Anne may be particularly interested in acquiring additional data to improve her model where it is weak. For e.g. this may be due to the fact that her original data may only cover a part of the feature space or distribution. Another example can be that her original data may have non-random missingness, which additional data may help with. It may also be useful to acquire more data points near the decision boundary etc.
I understand that the answers may vary depending on a lot of factors like the type of data, type of algorithm, the evaluation method, test distribution etc. But please feel free to make simplifying assumptions. The question is intentionally very general because I want to elicit answers from perspectives that I may not be aware of. You can also assume that Anne is indeed using the right model and the right learning algorithm, and there is scope to improve the model if she gets the right data.
 A: I don't have a "complete" answer, in the sense that I could describe the steps you could take in a pseudo-algorithmic to actualy solve the problem, but I might be able to point you in the right direction.
The problem you seem to describe sounds like Bayesian Experimental Design, and has connections with probabilistic machine learning and Bayesian optimazation.
The central idea in probabilistic machine learning is to take a Bayesian approach to doing statistical inference using models. This enables you to not only give a reasonably good point estimate $\hat y$ given some observation $x$, but also give you a notion of how uncertain the model is about this prediction (i.e. a statiscally valid error margin). If the thing you are trying to predict is very "close" to data you've observed before you would be more certain that such a prediction is correct as compared to a prediction for a point that is very far from all the data you've seen before.
What does this have to do with with your problem? Well, using Bayesian Experimental design you can find the "places" your model is most uncertain about and calculate the added utility if you were add data in this region, in terms of an experimental design. See: https://en.wikipedia.org/wiki/Bayesian_experimental_design
This idea is heavily used in Bayesian optimization of hyper parameters for instance, when tuning a ML model. Training new models over and over again, is very costly when the models are large. Therefor, making the best educated guess possible where to look for potential new model candidates is a very good idea. See: https://en.wikipedia.org/wiki/Bayesian_optimization
How would you use this to answer your question? Well, it would seem you could  probably calculate how much expected information is added when adding data from these new points (without the corresponding y values), as compared to say a random set, or the optimal set. Another idea is that, perhaps you could first calculate the points around which one would need to add more data to reduce the uncertainty in the model the most, and then calculate how "far" the datapoints in this new data set are from 1) the datapoints you already have and 2) the optimal datapoints which would have maximal utility in reducing model uncertainty.
I would look around the references from the wiki articles to see what you can find. I hope this at least points you in the right direction... Maybe somebody comes around which knows of a package/library/algorithm you can use to ease the calculation of what you're trying to do.
