This is probably a broad question, but I am interested to know if there are any studies that throw some light on this question:

If I have to train regression models (in the least-squared sense), how do various modeling techniques (e.g., OLS, L1/L2 regularised least-squares, neural networks, etc.) compare with respect to each other from a computational speed vs model quality (some measure of out-of-sample prediction error)?

Put another way, for a given data set, CPU resources, how much additional processing time would I need to train a broad class of neural networks to get the same/better performance as a simple OLS/L2 regularised regression?

  • $\begingroup$ I don't think it's realistic to seek an answer to this question quite the way you've asked it. After all, it's not as if there is a single, standard way to design each type of model -- OLS, NN, etc. To take advantage of the capabilities of each approach, one will want to work on model specification, which might involve transformed variables, interaction terms, penalization, etc. Obtaining good predictive accuracy requires a lot of thought and can seldom be achieved by rote application of tools. In spite of what some users of the SAS or SPSS data mining/modeler tools might say. $\endgroup$
    – rolando2
    Feb 8, 2018 at 19:10
  • $\begingroup$ If we were to abstract out the prediction process to finding a good estimator $f$ that minimises $(y - f(x))^2$, then there can be various model approaches to $f$ (OLS, NN, etc.), each with its own computation and performance tradeoffs. OLS with interaction terms, penalty, etc., can be thought of as a new modeling approach to $f$ that includes a broader class of functions, definitely requiring more computation, possibly with better performance. It is precisely that tradeoff that I would like to understand. $\endgroup$
    – Vimal
    Feb 8, 2018 at 21:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.