I am using multiple regression to predict the future value of a time series from several other time series. Would doing this with multilayer perceptrons produce better results than multiple regression?

Edit/expansion: Assuming a multiple regression equation behaves like a single perceptron, then as far as I understand it will not be able to cope with the XOR problem, and only uses a linear hyperplane to separate the data. This suggests to me that it can only deal with very simple relationships between the variables.

A multilayer perceptron, on the other hand, is a universal function approximater, and can cope with the XOR problem. But its disadvantage could be that of over-fitting, as it has many more internal variables than a multiple regression equation.

A remark elsewhere on this forum suggested that perceptions, and thus by analogy a multiple regression equation, could be tricked in to dealing with the XOR problem. Instead of having for example the equation (leaving out all the weightings) Y=A+B+C, use the equation Y=A+B+C+AB+BC+AC+ABC. I do not know if this would work.

The advantage of multiple regression over a perception is that the former is quick to calculate using standard statistical methods. A supplementary question is, could such a things as a multi-layered multiple regression equation(s) be created and used?

  • 4
    $\begingroup$ It's impossible to answer that a priori. $\endgroup$ – Firebug Jul 27 '17 at 15:10
  • 2
    $\begingroup$ @Firebug, why not write that up as an official answer? This seems like an intuitive question that people may have, & a short, informative, authoritative answer might be of substantial value. $\endgroup$ – gung Jul 28 '17 at 15:17

No Free Lunch!

Loosely stated, there is no best machine learning algorithm. An ML algorithm that does well in one type of problem "pays" for that superior performance by being worse on another set of problems. This is called the "No Free Lunch" theorem in machine learning. (There's a related theorem, with the same name, in search and optimization.)

A key paper on this topic is "The Lack of A Priori Distinctions Between Learning Algorithms" by David H. Wolpert Neural Computation 8, 1341-1390 (1996).

This is the first of two papers that use off-training set (OTS) error to investigate the assumption-free relationship between learning algorithms. This first paper discusses the senses in which there are no a priori distinctions between learning algorithms. (The second paper discusses the senses in which there are such distinctions.) In this first paper it is shown, loosely speaking, that for any two algorithms A and B, there are "as many" targets (or priors over targets) for which A has lower expected OTS error than B as vice versa, for loss functions like zero-one loss. In particular, this is true if A is cross-validation and B is "anti-cross-validation'' (choose the learning algorithm with largest cross-validation error). This paper ends with a discussion of the implications of these results for computational learning theory. It is shown that one cannot say: if empirical misclassification rate is low, the Vapnik-Chervonenkis dimension of your generalizer is small, and the training set is large, then with high probability your OTS error is small. Other implications for "membership queries" algorithms and "punting" algorithms are also discussed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.