I'm currently on a project where I basically need, like we all do, to understand how output y is related to input x. The particularity here is that data (y,x) is given to me one piece at a time, so I want to update my analysis every time I receive a new (y,x). I believe this is called "on-line" processing, as opposed to "batch" processing, where you have all the data needed and do your calculations using all data at the same time.
So I've looked around for ideas, and I've finally came with the conclusion that the world is divided in three:
The first part is the land of statistics and econometrics. People there do OLS, GLS, instrument variables, ARIMA, tests, difference of differences, PCA and whatnot. This land is mostly dominated by linearity and does only "batch" processing.
The second part is the island of machine learning and other words like artificial intelligence, supervised and unsupervised learning, neural networks and SVMs. Both "batch" and "on-line" processing are done here.
The third part is a whole continent I've just discovered, mostly populated by electrical engineers, so it seems. There, people often add the word "filter" to their tools, and they invented great stuffs like the Widrow-Hoff algorithm, Recursive least squares, the Wiener filter, the Kalman filter, and probably other things I haven't discovered yet. Apparently they do mostly "on-line" processing as it better fits their needs.
So my question is, do you have a global vision on all this? I'm under the impression that these three parts of the world don't talk too much to each other. Am I wrong? Is there a Grand Unified Theory of Understanding How Y Relates to X? Do you know any resources where the bases of that theory might be laid down?
I'm not sure if this question really makes sense, but I'm a little lost between all those theories. I imagine the answer to the question "should I use this or that?" would be "it depends on what you want to do (and on your data)". However I feel like those three worlds try to answer to the same question (y=f(x)?) and so it should be possible to have a higher view on all this, and deeply understand what makes each technique particular.