3
$\begingroup$

I have being working applying different ML Algorithms oriented to Big Data and I have some open questions that I find interesting to think about.

One of the first lectures about statistics begins with the sentence "Since the whole population data is not available, we must take representative samples so that our conclusions on this sample could be extended to the whole population." And Statistics is all about this.

However, Big Data allows not only access to the whole population but also the possibility of processing that information (something that was assumed false in statistics since the beginning).

So here are my questions:

  • If we can process all the data (the whole data), what's the point of doing nowadays cross-validation, sampling and the like?
  • If the answer to the latter question is 'overfitting', what if my algorithm is 'overfitted' with millions and millions and millions of data? In my opinion, if my algorithm is able to generalize that much I don't care if it is overfitted.
  • Do you think there are limitations concerning generalization for ML algorithms (SVMs, trees, Neural Networks) when the data is huge?
Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ By the way, I have seen a similar post here. That discussion answered my questions. $\endgroup$ – a.desantos Sep 14 '13 at 15:48
9
$\begingroup$

If you have access to "the whole data" it means that you already know for each input $x_i$ the desired output $y_i$, so you don't need any machine learning, you just answer $y_i$ once someone asks for $x_i$ (he cannot ask for $x$ such that $\neg \exists_i x_i = x$ as you assume that you already have all the data).

Big data is not about having "the whole data", and it does not contradict the statistical assumptions. Big data means exactly this: "we have lots of data in the comparison to the current computational power of the average computer". As a result, for many problems (in fact almost every problem) it still means that we do not know the whole data.

In general, big data does not change that much in machine learning as one could expect after reading all these papers with "big data" in the title. It is a well known phenomenon that with increased sample size our models require less and less regularization, as the underlying distributions are almost perfectly represented in the data. One can hear "the best machine learning algorithms are not based on the best models, but on the biggest datasets", but it's only partially true, as a large datasets does not mean a "good dataset", or even "unskewed". The amount of data can actually be a problem instead of help due to its hardly noticeable representations of rare phenomena.

Do you think there are limitations concerning generalization for ML algorithms (SVMs, trees, Neural Networks) when the data is huge?

Huge? What do you mean by huge? What do you mean by generalization limitations? This is to broad question. The generalization properties are very hard mathematical aspects of machine learning, and in my opinion cannot be addressed in such a "loose" manner.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I agree with you, but some times on the data there exist conflict information (non-congruent) and some ML algorithm may provide some light between that mess. What I meant by huge is a quantity large enough to require big-data technology (hadoop and the like) to process it. And by generalization I meant the capability of learning of an algorithm. $\endgroup$ – a.desantos Sep 14 '13 at 15:45
  • $\begingroup$ Requirement of using some technologies is far from any strict concept. And "capability of learning" is also far from strict. This is not a matter of claryfing used words, this is a matter of the too generic and broad question that cannot be addressed in this shape. $\endgroup$ – lejlot Sep 14 '13 at 15:52
  • $\begingroup$ One may wish to have a computationally less expensive model. Observing all of the data might give you a giant look-up table, but the cost of performing the look-up for a given $x_i$ is too great -- you need a simpler, compressed heuristic that calculates $y_i$ reasonably well and finishes faster than the look-up would. While that is not the main problem of statistical learning theory, it is an important one. $\endgroup$ – ely Oct 8 '13 at 18:53
  • $\begingroup$ @EMS This is exactly where massive parallelization techniques come to play, there is a nice recent book entitled "Scaling up Machine Learning: Parallel and Distributed Approaches" which deals with this problem $\endgroup$ – lejlot Oct 8 '13 at 19:02
  • $\begingroup$ I severely disagree. See, e.g. Amdahl's Law. It's much more of a fundamental complexity issue, something that parallelism cannot solve. If something is poly-time on a distributed system, then it's poly-time on a serial system with a big constant out front. You do not gain super-poly-time speedup from parallel programming. $\endgroup$ – ely Oct 8 '13 at 20:03
4
$\begingroup$

The answer to this hinges on the meaning of "the whole population". If you have data for the whole UK population as of 2013, or for all the rats in the lab last Tuesday; then for inference on the whole UK population as of 2013, or for all the rats in the lab last Tuesday, you don't need to cross-validate, or worry about over-fitting, &c. If you want to generalize to the UK population in 2016, to the French population; to Rattus norvegicus, or to mammals; then you do. Very often sampling from a population is a hypothetical rather than a real procedure; an (idealized) data-generating process is what's of interest.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, you are right. That is exactly what I meant. $\endgroup$ – a.desantos Sep 14 '13 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.