I'm currently on a project where I basically need, like we all do, to understand how output $y$ is related to input $x$. The particularity here is that data $(y,x)$ is given to me one piece at a time, so I want to update my analysis every time I receive a new $(y,x)$. I believe this is called "on-line" processing, as opposed to "batch" processing, where you have all the data needed and do your calculations using all data at the same time.

So I've looked around for ideas, and I've finally came with the conclusion that the world is divided in three:

  • The first part is the land of statistics and econometrics. People there do OLS, GLS, instrument variables, ARIMA, tests, difference of differences, PCA and whatnot. This land is mostly dominated by linearity and does only "batch" processing.

  • The second part is the island of machine learning and other words like artificial intelligence, supervised and unsupervised learning, neural networks and SVMs. Both "batch" and "on-line" processing are done here.

  • The third part is a whole continent I've just discovered, mostly populated by electrical engineers, so it seems. There, people often add the word "filter" to their tools, and they invented great stuffs like the Widrow-Hoff algorithm, Recursive least squares, the Wiener filter, the Kalman filter, and probably other things I haven't discovered yet. Apparently they do mostly "on-line" processing as it better fits their needs.

So my question is, do you have a global vision on all this? I'm under the impression that these three parts of the world don't talk too much to each other. Am I wrong? Is there a Grand Unified Theory of Understanding How $Y$ Relates to $X$? Do you know any resources where the bases of that theory might be laid down?

I'm not sure if this question really makes sense, but I'm a little lost between all those theories. I imagine the answer to the question "should I use this or that?" would be "it depends on what you want to do (and on your data)". However I feel like those three worlds try to answer to the same question ($y=f(x)$?) and so it should be possible to have a higher view on all this, and deeply understand what makes each technique particular.

  • $\begingroup$ I think the first 2 areas talk to each other a lot more these days. Great question though! $\endgroup$ – Zach Aug 21 '11 at 15:16
  • $\begingroup$ Dynamite topic and well written question! $\endgroup$ – rolando2 Aug 21 '11 at 18:04
  • 1
    $\begingroup$ Please make this CW. $\endgroup$ – cardinal Aug 21 '11 at 19:54
  • 1
    $\begingroup$ I'd call myself a statistician, but I do plenty of on-line stuff, do some nonlinear modelling of various kinds, and have dabbled in at least a little AI. I think the differences in typical tools has more to do with the kinds of problems people tend to encounter. Where their problems converge, sooner or later they tend to find or reinvent the same tools (often under different names and with slightly different bells and whistles). $\endgroup$ – Glen_b -Reinstate Monica Sep 16 '13 at 9:03

In terms of batch versus on-line , my experience tells me that sometimes you combine both. What I mean is that you let the heavy-lifting i.e compute intensive stuff relating to model formulation be done off-line and then employ quick/adaptive procedures to use these models. We have found that "new data" can be used in three ways; 1. to simply forecast ; 2. to revise parameters of known model and 3. to revise parameters and possibly revise model. These three approaches have been used for "live analysis" and of course the time to complete one of these three steps depends on both the software used and the hardware available.

Now to your other point regarding how to model y vs x. I prefer to use an expanded version of regression ( called Transfer Functions or ARMAX Models ) as the basis of pulling out the impact of the history of y and the current and pas values of x. It is critical that one validate the Gaussian requirements and to incorporate as necessary proxies for both omitted deterministic structure ( via Outlier Detection ) and omitted stochastic structure via the ARMA component. Additionally one needs to ensure that one hasn't used too much data ( tests for parameter constancy ) and that any non-constant error variance resulting from deterministic/stochastic error variance and/or linkage between the expected value of y and the variance of the residuals.

Now historically ( or hysterically if you wish ) different silos of thought have attempted to formulate approaches. Many of the ad-hoc models used by our ancestors cane be shown to be subsets of a Transfer Function but there are data sets that can be imagined which would challenge the assumptions of a Transfer Function. While these data sets may exist it should not be assumed that they will directly affect you unless analysis yields that conclusion.

Texts like Wei ( Addison-Wessley ) or Box-Jenkins should provide a reasonable road-map to support my commnents and to lead you to some more "answers"

By the way this is a Great Question !

Additionally if you have any data that you wish to use , I could demonstrate the various options outlined herein. Please post your data to the web for all to see and use in their efforts to relate "y to x" .

  • $\begingroup$ Thanks for your answer! I'll look more deeply into that as soon as I have time and probably I'll get back to you. I must say I didn't know the ARMAX model. I guess I'd have been directly to a all-endogenous VAR. As for the data, actually we're still building other stuffs for our project so I don't have a lot of relevant data right now. But thank you very much, you should hear from me again! $\endgroup$ – Arthur Aug 24 '11 at 21:22
  • $\begingroup$ "validate the Gaussian requirements": isn't Gaussian / nonparametric / forget-modelling (pace Breiman) a deep split ? $\endgroup$ – denis Aug 25 '11 at 8:04

Breiman address this issue in "Statistical Modeling: Two Cultures". A first response to an excellent question.

  • $\begingroup$ Thank! Your link didn't work for me, that one works and this one leads directly to the pdf. I've only read the abstract and some part randomly in the text and it looks very interesting. The guys seems to be totally "anti classic statistics", though. Thanks again. $\endgroup$ – Arthur Aug 24 '11 at 21:32
  • $\begingroup$ Great - I updated the link. It's an fun read -- enjoy! $\endgroup$ – Ram Ahluwalia Aug 24 '11 at 22:39
  • $\begingroup$ Breiman's "Two cultures" have been discussed here: some interesting points, but it's hard to change or even explain one's mind-set. $\endgroup$ – denis Aug 25 '11 at 14:59

I suspect the answer to this question is something along the lines of "there is no free lunch." Perhaps the reason statisticians, computer scientists, and electrical engineers have developed different algorithms is that they're interest in solving different sorts of problems.


I would say that these three group you indicated are indeed only two groups:

  • Statistics
  • Machine learning, artificial intelligence and pattern recognition.

All the branches related to signal filtering are based on two aspects: feature extraction (wavelets, Gabor and Fourier) which belongs to pattern recognition and Discrete Fourier Transformation which belongs to hard mathematics. In fact, digital filtering is more close to a engineering side as it try to solve this pattern recognition problem by means of simple and low computational cost algorithms. But essentially, it is machine learning.

Moreover, Filtering, Wavelets, Gabor and Fourier are extensively used in image processing being the core of artificial vision.

The difference exists between statistics and machine learning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.