I was reading the book Bayesian Methods for Hackers by Cameron Davidson-Pilon. He use PyMC
for examples.
As an experiment, I created a PySpark
App with the code example Inferring Behavior from Text-Message Data and I ran it on my computer with some network traffic as an standalone Spark cluster. It didn't take too long (382 ms for 27 MB of raw data). Now, I'm considering to scale this method to the full cluster and the hole data (24 MB per second).
So, my main concern is what do I have to consider for the implementation?
NOTE: In the book we can look at the following warning under A Note on Big Data
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?" ).
The much more difficult analytic problems involve medium data and, especially troublesome, really small data. Using a similar argument as Gelman's above, if big data problems are big enough to be readily solved, then we should be more interested in the not-quite-big enough datasets.
I thought Davidson-Pilon's warning about using Bayesian methods with big data was a problem reflected in the performance, but it isn't. So, what do I have to take in consideration when I apply Bayesian methods to a big data problem?
(I reviewed the references and actually I think my situation qualifies as a medium data problem.)