I have being working applying different ML Algorithms oriented to Big Data and I have some open questions that I find interesting to think about.
One of the first lectures about statistics begins with the sentence "Since the whole population data is not available, we must take representative samples so that our conclusions on this sample could be extended to the whole population." And Statistics is all about this.
However, Big Data allows not only access to the whole population but also the possibility of processing that information (something that was assumed false in statistics since the beginning).
So here are my questions:
- If we can process all the data (the whole data), what's the point of doing nowadays cross-validation, sampling and the like?
- If the answer to the latter question is 'overfitting', what if my algorithm is 'overfitted' with millions and millions and millions of data? In my opinion, if my algorithm is able to generalize that much I don't care if it is overfitted.
- Do you think there are limitations concerning generalization for ML algorithms (SVMs, trees, Neural Networks) when the data is huge?