Machine learning on big data: capability of generalization I have being working applying different ML Algorithms oriented to Big Data and I have some open questions that I find interesting to think about.
One of the first lectures about statistics begins with the sentence "Since the whole population data is not available, we must take representative samples so that our conclusions on this sample could be extended to the whole population." And Statistics is all about this.
However, Big Data allows not only access to the whole population but also the possibility of processing that information (something that was assumed false in statistics since the beginning).
So here are my questions:


*

*If we can process all the data (the whole data), what's the point of doing nowadays cross-validation, sampling and the like?

*If the answer to the latter question is 'overfitting', what if my algorithm is 'overfitted' with millions and millions and millions of data? In my opinion, if my algorithm is able to generalize that much I don't care if it is overfitted.

*Do you think there are limitations concerning generalization for ML algorithms (SVMs, trees, Neural Networks) when the data is huge? 

 A: If you have access to "the whole data" it means that you already know for each input $x_i$ the desired output $y_i$, so you don't need any machine learning, you just answer $y_i$ once someone asks for $x_i$ (he cannot ask for $x$ such that $\neg \exists_i x_i = x$ as you assume that you already have all the data).
Big data is not about having "the whole data", and it does not contradict the statistical assumptions. Big data means exactly this: "we have lots of data in the comparison to the current computational power of the average computer". As a result, for many problems (in fact almost every problem) it still means that we do not know the whole data.
In general, big data does not change that much in machine learning as one could expect after reading all these papers with "big data" in the title. It is a well known phenomenon that with increased sample size our models require less and less regularization, as the underlying distributions are almost perfectly represented in the data. One can hear "the best machine learning algorithms are not based on the best models, but on the biggest datasets", but it's only partially true, as a large datasets does not mean a "good dataset", or even "unskewed". The amount of data can actually be a problem instead of help due to its hardly noticeable representations of rare phenomena.

Do you think there are limitations concerning generalization for ML algorithms (SVMs, trees, Neural Networks) when the data is huge?

Huge? What do you mean by huge? What do you mean by generalization limitations? This is to broad question. The generalization properties are very hard mathematical aspects of machine learning, and in my opinion cannot be addressed in such a "loose" manner. 
A: The answer to this hinges on the meaning of "the whole population". If you have data for the whole UK population as of 2013, or for all the rats in the lab last Tuesday; then for inference on the whole UK population as of 2013, or for all the rats in the lab last Tuesday, you don't need to cross-validate, or worry about over-fitting, &c. If you want to generalize to the UK population in 2016, to the French population; to Rattus norvegicus, or to mammals; then you do. Very often sampling from a population is a hypothetical rather than a real procedure; an (idealized) data-generating process is what's of interest.
