Does machine learning really need data-efficient algorithms?

Question

Deep learning methods are often said to be very data-inefficient, requiring 100-1000 examples per class, where a human needs 1-2 to reach comparable classification accuracy.

However, modern datasets are huge (or can be made huge), which begs the question of whether we really need data-efficient algorithms. Are there application areas where a data-efficient machine learning algorithm would be very useful, despite making trade-offs elsewhere, e.g. training or inference efficiency? Would an ML algorithm that is, say, 100x more data-efficient, while being 1000x slower, be useful?

People who work on data-efficient algorithms often bring up robotics for "motivation". But even for robotics, large datasets can be collected, as is done in this data-collection factory at Google:

Basically, my concern is that while data-efficient algorithms exist (e.g. ILP, graphical models) and could be further improved, their practical applicability is squeezed between common tasks, where huge datasets exist, and rare ones, that may not be worth automating (leave something for humans!).

Answer 1

You are not entirely wrong, often it will be a lot easier to collect more/better data to improve an algorithm than to squeeze minor improvements out of the algorithm.

However, in practice, there are many settings, where it is difficult to get really large dataset.

Sure, it's easy to get really large datasets when you use (self-/un-)supervised approaches or if your labels are automatically created (e.g. if you are Google whether a user clicks on a link or not). However, many practical problems rely on human experts (whose time may be expensive) to label the examples. When any human can do the job (e.g. labeling dog or cat or something else for ImageNet), this can be scaled to millions of images, but when you pay physicians to classify medical images, tens of thousands (or perhaps 100,000ish) labelled images is a pretty large dataset. Or, if you need to run a chemical experiment for each label.

Additionally, there may be cases, where the or the number of possible real-world examples is naturally limited (e.g. training data for forecasting winners of US presidential elections, predicting eruptions of a volcanoes from seismic data etc., which are just things for which we so far can only have so much data).

Answer 2

I work in retail forecasting. When you need to forecast tomorrow's demand for product X at store Y, you only have a limited amount of data available: possibly only the last two years' worth of sales of this particular product at this particular store, or potentially sales of all products at all stores, if you use a cross-learning model. But in any case, you cannot simply create new data. (And creating new data consists in actually running your supermarket and recording sales and inventories, so this is not a trivial matter.)

Also, if a worldwide unprecedented pandemic hits you, the value of your data from before that time suddenly becomes dubious indeed, so for practical uses, your amount of data just decreased dramatically.

Of course, you are right that certain use cases have practically unlimited data, or can create data on the fly. One example is training networks to play games like chess or go: you can simply let multiple instances of your models play against each other (reinforcement learning).

Answer 3

While it is true that nowadays it is fairly easy to gather large piles of data, this doesn't mean that it is good data. The large datasets are usually gathered by scraping the resources freely available on Internet, for example, for textual data those may be Reddit post, news articles, Wikipedia entries, for images those may be all kinds of images posted by people, for videos those could be things posted on YouTube. Notice that there are many potential problems with such data.

First, it is unlabelled. To label it someone needs to do it. Most commonly, this is done by Amazon Mechanical Turk workers that are paid very little amounts of money for the task so aren't really motivated to do it correctly, neither have any internal motivation for tagging random images. Also, you have no guarantees that the labelers have proper knowledge for tagging (e.g. they are asked to label wild animals, or car brands, that they are not familiar with). You can do it yourself, but you would need a lot of time, and this as well doesn't guarantee that there won't be human errors. You could do the labeling automatically, but then your "clever" machine learning algorithm would learn from the labels provided by a "dumb" heuristic, if the heuristic worked, would you need the more complicated algorithm learn to imitate it..?

Second, this data is biased. Most of the textual datasets are limited to English languages. Most of the image datasets with photos of humans depict white-skinned individuals. Most of the datasets with pictures of architecture show cities from the US or Europe. Those aren't really representative, unless you are building a machine learning model that would be used only by the white, English-speaking men living in the US.

There was recently a nice preprint on this topic Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks by Northcutt et al.

Answer 4

Here are a couple thoughts to add to what has been posted so far.

You might be interested in taking a look at the famous machine learning paper, Domingos, P. (2012). "A Few Useful Things to Know about Machine Learning". Communications of the ACM (pdf). It should contain some food for thought. Specifically, here are three relevant subsections:

4. DATA ALONE IS NOT ENOUGH
   Generalization being the goal has another major consequence: data alone is not enough, no matter how much of it you have. Consider learning a Boolean function of (say) 100 variables from a million examples. There are $2^{100}$ − $10^6$ examples whose classes you don't know. How do you figure out what those classes are? In the absence of further information, there is just no way to do this that beats flipping a coin. ...

8. FEATURE ENGINEERING IS THE KEY
   At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used. If you have many independent features that each correlate well with the class, learning is easy. On the other hand, if the class is a very complex function of the features, you may not be able to learn it. Often, the raw data is not in a form that is amenable to learning, but you can construct features from it that are. ...

9. MORE DATA BEATS A CLEVERER ALGORITHM
   Suppose you’ve constructed the best set of features you can, but the classifiers you’re getting are still not accurate enough. What can you do now? There are two main choices: design a better learning algorithm, or gather more data (more examples, and possibly more raw features, subject to the curse of dimensionality). Machine learning researchers are mainly concerned with the former, but pragmatically the quickest path to success is often to just get more data. As a rule of thumb, a dumb algorithm with lots and lots of data beats a clever one with modest amounts of it. ...

The other thing I would say is that the idea that "a human needs 1-2 to reach comparable classification accuracy" is because the human is not a blank slate. A person has a wealth of experience (i.e., many prior data) and rich conceptual knowledge that can be brought to bear on learning a classification. (Sections 4 and 8 from Domingoes are related to this idea of background knowledge and knowing what to attend to.) To connect these facts to training a (deep learning or other) model, you could consider that pre-training a model sometimes helps quite a bit (although this is less done nowadays) and likewise that Bayesian models with sufficiently good priors should also perform better. Having said those things, section 9 from Domingoes implies we may be able to be sufficiently successful without those, due to the increasing volumes of data that you describe.

Answer 5

I was once asked to build a model that puts archeological artifacts into classes according to their manufactoring process. A big problem: for some classes, there were only four samples. And many artifacts are broken, so even for the samples we had, not all measurements were known (like their total length).

Yes, "small data" is indeed a problem. To get more data in this particular case would have meant to send the archeologists back to dig in the Central Asian mountains and to measure all the features of the artifacts that I find meaningful. At that, they better find artifacts in one piece, not broken ones! ;-)

Answer 6

There's some ambiguity in saying a data set is large. To improve predictive performance of an algorithm, you need more observations. You need to increase your sample size ($n$) and not the number of things you measured/observed within an experimental unit.

These can be hard to come by depending on the field of research: In clinical science there are privacy, security and most importantly ethical concerns with gathering more observations.

There are even cases where it is simply impossible to get more observations of independent experimental units. If you want to use prediction for some rare disease, or a nearly extinct species, there are hard constraints on how 'large' your data set can be.

Answer 7

Would an ML algorithm that is, say, 100x more data-efficient, while being 1000x slower, be useful?

You have almost answered your own question.

There are multiple factors at play here:

• The cost of gathering a data point
• The cost of training a model with an additional data point
• The cost of making the model learn more from a data point
• The benefit gained from training the model with an additional data point

You are seeking to maximize the expression (benefits - costs). If you measure or estimate these factors accurately enough, and convert to comparable units (such as monetary equivalents perhaps), you'll find it easy to determine what to improve most easily.

As others have said, there are various applications with completely different such factors.

Answer 8

People who work on data-efficient algorithms often bring up robotics for "motivation". But even for robotics, large datasets can be collected, as is done in this data-collection factory at Google:

What if I want to (for example) use reinforcement learning on a task involving underwater robotics to classify arctic ocean fronts? Or train a vision module to classify extremely rare objects in space through a fly-by probe? I may have very limited data and the cost of gathering new data may be extremely expensive. Often in robotics simulators are not accurate enough (especially when natural phenonmena are involved) to really generate accurate training data (this is called the sim2real problem).

Additionally, gathering real-life data for every possible task you would like your robot to accomplish can be prohibitive, especially if you want a wide variety of tasks accomplished by something like an in-home robot.

Answer 9

To all the other answers I'd add that in Deep Learning, Neural Architecture Search benefits immensely from data efficiency. Think about it: each data point is a trained network.

If your NAS setup requires $N$ data points (networks), and each network requires $D$ samples to be trained, that's $ND$ forward- and backpropagations overall: if you reduce $D$ by a ratio of $k$, that's $k$-many more architectures that you can explore with the same resources.

Of course, it isn't always that straightforward: This CVPR2021 paper by Mundt et al shows that the architectures themselves act as Deep Priors which don't need to be fully trained, if initialized correctly (as a nice counterpoint to the vast few-shot learning literature):

Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

Answer 10

To generalize a bit what @FransrRodenburg says about sample size: many data sets have structure from various influencing factors. In particular, there are certain situation that lead to what is called nested factors in statistics (clustered data sets, hierarchical data sets, 1 : n relationships between influencing factors) and in this situation the overall sample size is often very limited, while data is abundant at lower levels in the data hierarchy.

I'm spectroscopist and chemometrician, here are some examples:

• I did my PhD about classification of brain tissues for tumor (boundary) recognition.

  • I got hundreds of spectra from each piece of tissue,
  • but the number of patients were only order of magnitude 100.
  • For one of the tumor types I was looking at, primary lymphomas of the central nervous system, surgery is only done if the bulk of the tumor causes trouble. After 7 years, our cooperation partners had collected 8 pieces from 5 patients. (For comparison: during the same time, samples of about 2000 glioblastomas were collected for us) The application scenario btw was guiding a biopsy needle, which needs to be done far more often - but one wouldn't remove additional brain tissue samples for research purposes there.
    BTW: we had a huge problem finding good controls (normal brain tissue). Guess why.

• I have a data set of about 8 TB hyperspectral imaging data containing a few thousand cacao beans collected at various stages of fermentation (they are classified according to color), from a few varieties, from 4 regions. And exactly n = 1 harvest year.

  Yes, there are companies who collect large data sets, data bases covering, thousands of farms on several continents. Still AFAIK the largest have "only" a few decades of harvesting periods...

---

rare ones, that may not be worth automating (leave something for humans!).

• The difficulty of getting samples (I'm talking of the physical pieces of material) does not necessarily say much about the need or the economic potential of the automated application. See e.g. the tumour example

• During that tumor work I met a bunch of pathologists with whom I talked about their needs.

  • Some of them expressed needs rougly along your lines: "If I had an instrument that would automatically deal with the 95 % easy samples, I could concentrate on the rare/special/complicated/interesting cases"
  • Others were thinking pretty much of the opposite: "I can get through a routine cases within seconds - it's the rare/special/complicated/interesting cases where I'd really appreciate additional information."

  (side note: one of the most important mistakes I see regularly is that the application scenario isn't specified in sufficient detail, resulting in something that mixes e.g. the two scenarios above and in the end doesn't meet the requirements of anyone.)

Answer 11

Random thoughts, although does not fully answer the question.

There seems to be a wastage of information in training new models on new data even if it is plentiful. Using an analogy with another general purpose technology, fitting new models is not totally dissimilar to reinventing the wheel. Bayesian and transfer learning seem to offer solutions in adding to cumulative data and knowledge, and hence to an extent can help mitigate this. Would the problem of replication crisis be as deep if Bayesian techniques had been used more and thus data containing surprisng results had to overcome the inertia of previous studies?

As @svavil highlights, it is the accumulation of prior training that has accumulated in the past that allows (seemingly?) impressive results in the present. Training efficiency can be obtained by incorporating new data on top of previous data by transfer learning, and neural networks because of their usage in many different domains seem to be amenable to transfer learning (is this right?) and transfer learning is sometimes (usually?) going to mean potentially some efficiencies. Further aggregating of different neural networks seems to also mean potential efficiency of say training a lidar data set by a neural network and then concatenating (word used in a non-technical way!) this neural network to a radar trained neural network.

As per the answer by @Tim, biased data can be a problem, and reusing a known good data set, say as a starting point for transfer learning may be helpful in mitigating a bias problem. As an aside, biased data presumably increases the problem if you accidently skew the bias-variance tradeoff too much for the former (low bias, high variance), and thus generalization will be even worse. Note here there are two types of bias used in the above sentence, and feel free to comment on this although is just an aside here.

So the question may be able to be reversed, techniques that add to accumulated knowledge are helpful, and they also tend to be efficient when new data is added to them.