Suppose I have massive data set (think Terabytes) is available to train a learning algorithm. Which one of the following conditions must be true to obtain good performance (low error rate)

a. Using a complex model with many parameters, including high order polynomial features of the original space (e.g x^10)

b. Training with a small number of parameters

I am thinking about regularization by adding peanlty and Bias -variance tradeoff. Please suggest what is good way to go...

  • $\begingroup$ If you have enough data to do sufficient cross-validation, I'm not sure it trying to use more parameters will hurt you or lead to over-fitting. $\endgroup$ Commented Oct 21, 2012 at 20:12

1 Answer 1


What is most important is understaning the science behind the data. If you add enough nonsense variables to your model you can predict pretty much any variable, but it will not help much for a new data set.

If you have 2 predictors that are correlated with each other then an interaction between the 2 will be fairly similar to a quadratic term on either, looking at only the data cannot tell you which is the most meaningful, you need to understand the nature of the process that generated the data to make these decisions.

Also your goals can influence things, are you trying to create a model to predict future outcomes? Are you trying to understand if certain variables may cause the response even after adjusting for others? or something else? which is the best model can be different for different goals.

  • $\begingroup$ Yes i am trying to check which one of the methods will be efficienct with learning algorithm x In order to lower the error Polynomial linear regression with regularization Or the other one training with less parameters $\endgroup$
    – user16106
    Commented Oct 20, 2012 at 18:31
  • $\begingroup$ @user16106 Have you lost your account information? It looks like you registered twice. $\endgroup$
    – chl
    Commented Oct 20, 2012 at 18:49
  • $\begingroup$ No I logged in from my spouse's mobile so it took that.. $\endgroup$
    – user16096
    Commented Oct 20, 2012 at 19:32
  • $\begingroup$ @user16096 Do we have to merge them? $\endgroup$
    – chl
    Commented Oct 21, 2012 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.