In what type of application a small training set would be important? I have always heard that the more the samples for training a model the better.
But a lot of samples means a lot of computation time while creating the model. 
Is there any application where using few samples for training a model, is justified? Is it a good practice? 
Thank you
 A: Generally speaking, the more data is available, the better. In fact, papers have found that training a model on 300 million images is better than only 30 million -- it's not the case that our models can be fully "saturated" with just a few million images.
However, obtaining ground truth labels for millions of images can be very expensive. Increasing the sample efficiency of models is a very active area of research. For example, there is lots of work in classifying images with only one sample.
The best approach, from a pragmatic point of view, is to gather as much data as possible, since it generally leads to better results.
A: Although this is the type of question where the most correct answer is probably "it depends," I will provide an answer by highlighting an application where a small number of samples is used for training.
Data streams are a challenging learning environment. Faria et al. offer the following definition (based off of Aggarwal et al.'s work):

A data stream $S$ is a massive sequence of multi-dimensional examples $x_1, x_2,..., x_N,…$, which is potentially unbounded, arriving in time stamps $T_1,T_2,...,T_N,…$ Each example $x_i$ is described by an n-dimensional attribute vector.

One specifically challenging aspect of data streams is that they can represent non-stationary environments: the underlying probability distributions from which the instances are drawn and what constitutes 'normal behaviour' can change over time. This is called "concept drift."
With the (potentially) infinite size of the data stream, it is not possible to keep all of the instances in memory. Therefore, right from the start, the algorithm must consider a limited number of instances. Without getting too far into the literature on concept drifting data streams, there are a few aspects which illustrate this (from Gama et al.):


*

*Approaches that retrain do so by discarding their old model and building a completely new one on the basis of the instances being received from the data stream;

*Approaches that increment do so by combining their model and/or a summary of past data with what can be learned from the new data;

*Approaches that update their model based on change detection do so by collecting a number of instances from the newly detected concept before training; and

*Approaches that use a sliding window as a forgetting mechanism inherently limit themselves to considering a limited number of instances from the data stream.


In each of these scenarios, the ideal size of the window/number of instances to consider is variable, but a common consideration is that it should be small enough to allow the algorithm to react to concept drift in a timely manner (while being large enough to be stable when there is no concept drift).
To summarize, the time dimension of the data stream environment produces a pressure on algorithms to be able to learn from a small number of instances (at least initially) in order to adapt to a non-stationary environment in a timely manner.
