20
$\begingroup$

Deep learning methods are often said to be very data-inefficient, requiring 100-1000 examples per class, where a human needs 1-2 to reach comparable classification accuracy.

However, modern datasets are huge (or can be made huge), which begs the question of whether we really need data-efficient algorithms. Are there application areas where a data-efficient machine learning algorithm would be very useful, despite making trade-offs elsewhere, e.g. training or inference efficiency? Would an ML algorithm that is, say, 100x more data-efficient, while being 1000x slower, be useful?

People who work on data-efficient algorithms often bring up robotics for "motivation". But even for robotics, large datasets can be collected, as is done in this data-collection factory at Google:

enter image description here

Basically, my concern is that while data-efficient algorithms exist (e.g. ILP, graphical models) and could be further improved, their practical applicability is squeezed between common tasks, where huge datasets exist, and rare ones, that may not be worth automating (leave something for humans!).

$\endgroup$
19
  • 24
    $\begingroup$ Frame challenge: How would you gather large amounts of data for a rare disease? Or how would you justify using more model organisms, let alone patients, when you can do with less? $\endgroup$ – Frans Rodenburg Jun 9 at 8:49
  • 4
    $\begingroup$ In medical imaging, the set of all existing and available at some point images (e.g. CT, MR, PET, etc) is smaller than the number of "cats" in a single classification algorithm. We could inject radioactivity to the entire world population, and have them 1h each in the scan, to gather enough PET images for training.... or could we? $\endgroup$ – Ander Biguri Jun 9 at 9:09
  • 14
    $\begingroup$ A human neural network can solve a classification problem after seeing 1-2 examples because it underwent extensive prior learning on unrelated datasets. $\endgroup$ – svavil Jun 9 at 13:36
  • 4
    $\begingroup$ @Firebug I am a bit confused really. I do not understand how the existence of few populous datasets invalidates my original comment, so I don't understand why you shared that. If you want a counter example, can you find me dynamic PET scans? 15 would do. Also note: that dataset (as most, if not all available medical ones that exist) only provide reconstructed data, its already been processed. Raw data is often much more interesting and the machines themselves do not output it to the clinicians. $\endgroup$ – Ander Biguri Jun 9 at 13:51
  • 7
    $\begingroup$ Noticing that large datasets can be collected even for robotics struck me as sort of funny. A maximally absurd way of restating your final paragraph would be, like, "If I can rent 100 robots for 1000 hours, why would I want to improve my algorithm and rent 1 robot for 10 hours?". Projects that build successful ML applications have a large chunk of their budget set aside for gathering data, sometimes millions or tens of millions of dollars. If you can make that cost thousands or tens of thousands of dollars instead, they're going to have millions left over to throw at buying your algorithm. $\endgroup$ – Daniel Wagner Jun 9 at 21:52

11 Answers 11

30
$\begingroup$

You are not entirely wrong, often it will be a lot easier to collect more/better data to improve an algorithm than to squeeze minor improvements out of the algorithm.

However, in practice, there are many settings, where it is difficult to get really large dataset.

Sure, it's easy to get really large datasets when you use (self-/un-)supervised approaches or if your labels are automatically created (e.g. if you are Google whether a user clicks on a link or not). However, many practical problems rely on human experts (whose time may be expensive) to label the examples. When any human can do the job (e.g. labeling dog or cat or something else for ImageNet), this can be scaled to millions of images, but when you pay physicians to classify medical images, tens of thousands (or perhaps 100,000ish) labelled images is a pretty large dataset. Or, if you need to run a chemical experiment for each label.

Additionally, there may be cases, where the or the number of possible real-world examples is naturally limited (e.g. training data for forecasting winners of US presidential elections, predicting eruptions of a volcanoes from seismic data etc., which are just things for which we so far can only have so much data).

$\endgroup$
5
  • 2
    $\begingroup$ Your examples at the end seem like problems where ML may not be an appropriate approach, and may not be amenable to AI at all. Humans also can't forecast elections or eruptions. If a goal of AI is to mimic human capabilities, you're already there. $\endgroup$ – Barmar Jun 9 at 14:29
  • 13
    $\begingroup$ However you judge the work of fivethirtyeight.com they certainly try to forecast elections and from my perspective do a pretty decent job. Sure, they take a very Bayesian approach, which is not necessarily the current ML mainstream, but by most definitions what they do is still ML or AI just encoding as much human prior knowledge as possible. $\endgroup$ – Björn Jun 9 at 14:48
  • $\begingroup$ @Björn Aren't Bayesian approaches by definition much better at incorporating human domain expertise than deep learning approaches, and by virtue of that require less data to perform well? The original question seems entirely framed around the data-inefficient deep-learning ML approach. $\endgroup$ – gmatt Jun 10 at 19:07
  • $\begingroup$ @gmatt, that is one of the big attractions of Bayesian approaches, but they are sometimes also just better at representing uncertainty (or even at getting something like CI/CrI) in small sample situations than frequentist methods, even if you do not have much prior information. Representing uncertainty is also - I believe - the driver behind the interest in Bayesian deep learning (see e.g. the work by Yarin Gal e.g. cs.ox.ac.uk/activities/bayesiandeeplearning). It seemed to me that the question was sort of questioning whether we ever need anything else than DL (+lots of data). $\endgroup$ – Björn Jun 10 at 21:46
  • 2
    $\begingroup$ @gmatt Doesn't that make it a good answer to the question then? The data inefficient method doesn't work great and an alternative data efficient method does, which is a good motivation to have the data efficient method available. $\endgroup$ – mbrig Jun 11 at 0:07
26
$\begingroup$

I work in retail forecasting. When you need to forecast tomorrow's demand for product X at store Y, you only have a limited amount of data available: possibly only the last two years' worth of sales of this particular product at this particular store, or potentially sales of all products at all stores, if you use a cross-learning model. But in any case, you cannot simply create new data. (And creating new data consists in actually running your supermarket and recording sales and inventories, so this is not a trivial matter.)

Also, if a worldwide unprecedented pandemic hits you, the value of your data from before that time suddenly becomes dubious indeed, so for practical uses, your amount of data just decreased dramatically.

Of course, you are right that certain use cases have practically unlimited data, or can create data on the fly. One example is training networks to play games like chess or go: you can simply let multiple instances of your models play against each other (reinforcement learning).

$\endgroup$
5
  • 1
    $\begingroup$ Electricity forecasting at meter level (i.e. per building) is very similar. We typically only have 1 year of data per site - usually not the same year for all sites. That's only 1 seasonal cycle, and the energy consumption is fairly non-stationary, particularly as you say due to covid. $\endgroup$ – David Waterworth Jun 11 at 9:22
  • $\begingroup$ @DavidWaterworth: I fully agree. That said, what would be the use case for forecasting at the meter level? In forecasting supermarket sales, a product that lies on the shelf at store A is not available to a shopper at store B, but there is no analogue for electricity that I can see, so I don't quite see why one would want to forecast electricity consumption at this granular level. Do you have an example? $\endgroup$ – Stephan Kolassa Jun 11 at 9:27
  • $\begingroup$ By meter level, I mean by building. It's quite common, usually to create a baseline in order to measure the impact of some energy efficiency program (i.e. the model is a counterfactual of what building energy demand would have been had you not installed energy-efficient lights for example) $\endgroup$ – David Waterworth Jun 11 at 9:34
  • $\begingroup$ @DavidWaterworth: good point - thanks! $\endgroup$ – Stephan Kolassa Jun 11 at 9:43
  • $\begingroup$ At the abstract level, its "the world changes so we only ever have a limited amount of relevant data", and thus need data efficient algorithms. $\endgroup$ – Dave Jun 11 at 15:08
11
$\begingroup$

While it is true that nowadays it is fairly easy to gather large piles of data, this doesn't mean that it is good data. The large datasets are usually gathered by scraping the resources freely available on Internet, for example, for textual data those may be Reddit post, news articles, Wikipedia entries, for images those may be all kinds of images posted by people, for videos those could be things posted on YouTube. Notice that there are many potential problems with such data.

First, it is unlabelled. To label it someone needs to do it. Most commonly, this is done by Amazon Mechanical Turk workers that are paid very little amounts of money for the task so aren't really motivated to do it correctly, neither have any internal motivation for tagging random images. Also, you have no guarantees that the labelers have proper knowledge for tagging (e.g. they are asked to label wild animals, or car brands, that they are not familiar with). You can do it yourself, but you would need a lot of time, and this as well doesn't guarantee that there won't be human errors. You could do the labeling automatically, but then your "clever" machine learning algorithm would learn from the labels provided by a "dumb" heuristic, if the heuristic worked, would you need the more complicated algorithm learn to imitate it..?

Second, this data is biased. Most of the textual datasets are limited to English languages. Most of the image datasets with photos of humans depict white-skinned individuals. Most of the datasets with pictures of architecture show cities from the US or Europe. Those aren't really representative, unless you are building a machine learning model that would be used only by the white, English-speaking men living in the US.

There was recently a nice preprint on this topic Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks by Northcutt et al.

$\endgroup$
10
$\begingroup$

I was once asked to build a model that puts archeological artifacts into classes according to their manufactoring process. A big problem: for some classes, there were only four samples. And many artifacts are broken, so even for the samples we had, not all measurements were known (like their total length).

Yes, "small data" is indeed a problem. To get more data in this particular case would have meant to send the archeologists back to dig in the Central Asian mountains and to measure all the features of the artifacts that I find meaningful. At that, they better find artifacts in one piece, not broken ones! ;-)

New contributor
user33593 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
10
$\begingroup$

Here are a couple thoughts to add to what has been posted so far.

You might be interested in taking a look at the famous machine learning paper, Domingos, P. (2012). "A Few Useful Things to Know about Machine Learning". Communications of the ACM (pdf). It should contain some food for thought. Specifically, here are three relevant subsections:

  1. DATA ALONE IS NOT ENOUGH
    Generalization being the goal has another major consequence: data alone is not enough, no matter how much of it you have. Consider learning a Boolean function of (say) 100 variables from a million examples. There are $2^{100}$ ~ $10^6$ examples whose classes you don’t know. How do you figure out what those classes are? In the absence of further information, there is just no way to do this that beats flipping a coin. ...
  1. FEATURE ENGINEERING IS THE KEY
    At the end of the day, some machine learning projects suc- ceed and some fail. What makes the difference? Easily the most important factor is the features used. If you have many independent features that each correlate well with the class, learning is easy. On the other hand, if the class is a very complex function of the features, you may not be able to learn it. Often, the raw data is not in a form that is amenable to learning, but you can construct features from it that are. ...
  1. MORE DATA BEATS A CLEVERER ALGORITHM
    Suppose you’ve constructed the best set of features you can, but the classifiers you’re getting are still not accurate enough. What can you do now? There are two main choices: design a better learning algorithm, or gather more data (more examples, and possibly more raw features, subject to the curse of dimensionality). Machine learning researchers are mainly concerned with the former, but pragmatically the quickest path to success is often to just get more data. As a rule of thumb, a dumb algorithm with lots and lots of data beats a clever one with modest amounts of it. ...

The other thing I would say is that the idea that "a human needs 1-2 to reach comparable classification accuracy" is because the human is not a blank slate. A person has a wealth of experience (i.e., many prior data) and rich conceptual knowledge that can be brought to bear on learning a classification. (Sections 4 and 8 from Domingoes are related to this idea of background knowledge and knowing what to attend to.) To connect these facts to training a (deep learning or other) model, you could consider that pre-training a model sometimes helps quite a bit (although this is less done nowadays) and likewise that Bayesian models with sufficiently good priors should also perform better. Having said those things, section 9 from Domingoes implies we may be able to be sufficiently successful without those, due to the increasing volumes of data that you describe.

$\endgroup$
1
  • 3
    $\begingroup$ Curiously, Pedro's math is wrong here: $2^{100}$ is approximately $10^{30}$, not $10^6$ $\endgroup$ – bobcat Jun 9 at 20:18
4
$\begingroup$

There's some ambiguity in saying a data set is large. To improve predictive performance of an algorithm, you need more observations. You need to increase your sample size ($n$) and not the number of things you measured/observed within an experimental unit.

These can be hard to come by depending on the field of research: In clinical science there are privacy, security and most importantly ethical concerns with gathering more observations.

There are even cases where it is simply impossible to get more observations of independent experimental units. If you want to use prediction for some rare disease, or a nearly extinct species, there are hard constraints on how 'large' your data set can be.

$\endgroup$
2
$\begingroup$

Would an ML algorithm that is, say, 100x more data-efficient, while being 1000x slower, be useful?

You have almost answered your own question.

There are multiple factors at play here:

  • The cost of gathering a data point
  • The cost of training a model with an additional data point
  • The cost of making the model learn more from a data point
  • The benefit gained from training the model with an additional data point

You are seeking to maximize the expression (benefits - costs). If you measure or estimate these factors accurately enough, and convert to comparable units (such as monetary equivalents perhaps), you'll find it easy to determine what to improve most easily.

As others have said, there are various applications with completely different such factors.

$\endgroup$
2
$\begingroup$

People who work on data-efficient algorithms often bring up robotics for "motivation". But even for robotics, large datasets can be collected, as is done in this data-collection factory at Google:

What if I want to (for example) use reinforcement learning on a task involving underwater robotics to classify arctic ocean fronts? Or train a vision module to classify extremely rare objects in space through a fly-by probe? I may have very limited data and the cost of gathering new data may be extremely expensive. Often in robotics simulators are not accurate enough (especially when natural phenonmena are involved) to really generate accurate training data (this is called the sim2real problem).

Additionally, gathering real-life data for every possible task you would like your robot to accomplish can be prohibitive, especially if you want a wide variety of tasks accomplished by something like an in-home robot.

$\endgroup$
1
$\begingroup$

To all the other answers I'd add that in Deep Learning, Neural Architecture Search benefits immensely from data efficiency. Think about it: each data point is a trained network.

If your NAS setup requires $N$ data points (networks), and each network requires $D$ samples to be trained, that's $ND$ forward- and backpropagations overall: if you reduce $D$ by a ratio of $k$, that's $k$-many more architectures that you can explore with the same resources.

Of course, it isn't always that straightforward: This CVPR2021 paper by Mundt et al shows that the architectures themselves act as Deep Priors which don't need to be fully trained, if initialized correctly (as a nice counterpoint to the vast few-shot learning literature):

Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

$\endgroup$
0
1
$\begingroup$

To generalize a bit what @FransrRodenburg says about sample size: many data sets have structure from various influencing factors. In particular, there are certain situation that lead to what is called nested factors in statistics (clustered data sets, hierarchical data sets, 1 : n relationships between influencing factors) and in this situation the overall sample size is often very limited, while data is abundant at lower levels in the data hierarchy.

I'm spectroscopist and chemometrician, here are some examples:

  • I did my PhD about classification of brain tissues for tumor (boundary) recognition.

    • I got hundreds of spectra from each piece of tissue,
    • but the number of patients were only order of magnitude 100.
    • For one of the tumor types I was looking at, primary lymphomas of the central nervous system, surgery is only done if the bulk of the tumor causes trouble. After 7 years, our cooperation partners had collected 8 pieces from 5 patients. (For comparison: during the same time, samples of about 2000 glioblastomas were collected for us) The application scenario btw was guiding a biopsy needle, which needs to be done far more often - but one wouldn't remove additional brain tissue samples for research purposes there.
      BTW: we had a huge problem finding good controls (normal brain tissue). Guess why.
  • I have a data set of about 8 TB hyperspectral imaging data containing a few thousand cacao beans collected at various stages of fermentation (they are classified according to color), from a few varieties, from 4 regions. And exactly n = 1 harvest year.

    Yes, there are companies who collect large data sets, data bases covering, thousands of farms on several continents. Still AFAIK the largest have "only" a few decades of harvesting periods...


rare ones, that may not be worth automating (leave something for humans!).

  • The difficulty of getting samples (I'm talking of the physical pieces of material) does not necessarily say much about the need or the economic potential of the automated application. See e.g. the tumour example

  • During that tumor work I met a bunch of pathologists with whom I talked about their needs.

    • Some of them expressed needs rougly along your lines: "If I had an instrument that would automatically deal with the 95 % easy samples, I could concentrate on the rare/special/complicated/interesting cases"
    • Others were thinking pretty much of the opposite: "I can get through a routine cases within seconds - it's the rare/special/complicated/interesting cases where I'd really appreciate additional information."

    (side note: one of the most important mistakes I see regularly is that the application scenario isn't specified in sufficient detail, resulting in something that mixes e.g. the two scenarios above and in the end doesn't meet the requirements of anyone.)

$\endgroup$
0
$\begingroup$

Random thoughts, although does not fully answer the question.

There seems to be a wastage of information in training new models on new data even if it is plentiful. Using an analogy with another general purpose technology, fitting new models is not totally dissimilar to reinventing the wheel. Bayesian and transfer learning seem to offer solutions in adding to cumulative data and knowledge, and hence to an extent can help mitigate this. Would the problem of replication crisis be as deep if Bayesian techniques had been used more and thus data containing surprisng results had to overcome the inertia of previous studies?

As @svavil highlights, it is the accumulation of prior training that has accumulated in the past that allows (seemingly?) impressive results in the present. Training efficiency can be obtained by incorporating new data on top of previous data by transfer learning, and neural networks because of their usage in many different domains seem to be amenable to transfer learning (is this right?) and transfer learning is sometimes (usually?) going to mean potentially some efficiencies. Further aggregating of different neural networks seems to also mean potential efficiency of say training a lidar data set by a neural network and then concatenating (word used in a non-technical way!) this neural network to a radar trained neural network.

As per the answer by @Tim, biased data can be a problem, and reusing a known good data set, say as a starting point for transfer learning may be helpful in mitigating a bias problem. As an aside, biased data presumably increases the problem if you accidently skew the bias-variance tradeoff too much for the former (low bias, high variance), and thus generalization will be even worse. Note here there are two types of bias used in the above sentence, and feel free to comment on this although is just an aside here.

So the question may be able to be reversed, techniques that add to accumulated knowledge are helpful, and they also tend to be efficient when new data is added to them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.