Do you have a global vision on those analysis techniques?

Question

I'm currently on a project where I basically need, like we all do, to understand how output $y$ is related to input $x$. The particularity here is that data $(y,x)$ is given to me one piece at a time, so I want to update my analysis every time I receive a new $(y,x)$. I believe this is called "on-line" processing, as opposed to "batch" processing, where you have all the data needed and do your calculations using all data at the same time.

So I've looked around for ideas, and I've finally came with the conclusion that the world is divided in three:

• The first part is the land of statistics and econometrics. People there do OLS, GLS, instrument variables, ARIMA, tests, difference of differences, PCA and whatnot. This land is mostly dominated by linearity and does only "batch" processing.

• The second part is the island of machine learning and other words like artificial intelligence, supervised and unsupervised learning, neural networks and SVMs. Both "batch" and "on-line" processing are done here.

• The third part is a whole continent I've just discovered, mostly populated by electrical engineers, so it seems. There, people often add the word "filter" to their tools, and they invented great stuffs like the Widrow-Hoff algorithm, Recursive least squares, the Wiener filter, the Kalman filter, and probably other things I haven't discovered yet. Apparently they do mostly "on-line" processing as it better fits their needs.

So my question is, do you have a global vision on all this? I'm under the impression that these three parts of the world don't talk too much to each other. Am I wrong? Is there a Grand Unified Theory of Understanding How $Y$ Relates to $X$? Do you know any resources where the bases of that theory might be laid down?

I'm not sure if this question really makes sense, but I'm a little lost between all those theories. I imagine the answer to the question "should I use this or that?" would be "it depends on what you want to do (and on your data)". However I feel like those three worlds try to answer to the same question ($y=f(x)$?) and so it should be possible to have a higher view on all this, and deeply understand what makes each technique particular.

Answer 1

In terms of batch versus on-line , my experience tells me that sometimes you combine both. What I mean is that you let the heavy-lifting i.e compute intensive stuff relating to model formulation be done off-line and then employ quick/adaptive procedures to use these models. We have found that "new data" can be used in three ways; 1. to simply forecast ; 2. to revise parameters of known model and 3. to revise parameters and possibly revise model. These three approaches have been used for "live analysis" and of course the time to complete one of these three steps depends on both the software used and the hardware available.

Now to your other point regarding how to model y vs x. I prefer to use an expanded version of regression ( called Transfer Functions or ARMAX Models ) as the basis of pulling out the impact of the history of y and the current and pas values of x. It is critical that one validate the Gaussian requirements and to incorporate as necessary proxies for both omitted deterministic structure ( via Outlier Detection ) and omitted stochastic structure via the ARMA component. Additionally one needs to ensure that one hasn't used too much data ( tests for parameter constancy ) and that any non-constant error variance resulting from deterministic/stochastic error variance and/or linkage between the expected value of y and the variance of the residuals.

Now historically ( or hysterically if you wish ) different silos of thought have attempted to formulate approaches. Many of the ad-hoc models used by our ancestors cane be shown to be subsets of a Transfer Function but there are data sets that can be imagined which would challenge the assumptions of a Transfer Function. While these data sets may exist it should not be assumed that they will directly affect you unless analysis yields that conclusion.

Texts like Wei ( Addison-Wessley ) or Box-Jenkins should provide a reasonable road-map to support my commnents and to lead you to some more "answers"

By the way this is a Great Question !

Additionally if you have any data that you wish to use , I could demonstrate the various options outlined herein. Please post your data to the web for all to see and use in their efforts to relate "y to x" .

Answer 2

Breiman address this issue in "Statistical Modeling: Two Cultures". A first response to an excellent question.

Answer 3

I suspect the answer to this question is something along the lines of "there is no free lunch." Perhaps the reason statisticians, computer scientists, and electrical engineers have developed different algorithms is that they're interest in solving different sorts of problems.

Answer 4

I would say that these three group you indicated are indeed only two groups:

• Statistics
• Machine learning, artificial intelligence and pattern recognition.

All the branches related to signal filtering are based on two aspects: feature extraction (wavelets, Gabor and Fourier) which belongs to pattern recognition and Discrete Fourier Transformation which belongs to hard mathematics. In fact, digital filtering is more close to a engineering side as it try to solve this pattern recognition problem by means of simple and low computational cost algorithms. But essentially, it is machine learning.

Moreover, Filtering, Wavelets, Gabor and Fourier are extensively used in image processing being the core of artificial vision.

The difference exists between statistics and machine learning.