How to obtain good performance (low error rate) on massive data set?

Question

Suppose I have massive data set (think Terabytes) is available to train a learning algorithm. Which one of the following conditions must be true to obtain good performance (low error rate)

a. Using a complex model with many parameters, including high order polynomial features of the original space (e.g x^10)

b. Training with a small number of parameters

I am thinking about regularization by adding peanlty and Bias -variance tradeoff. Please suggest what is good way to go...

Answer 1

What is most important is understaning the science behind the data. If you add enough nonsense variables to your model you can predict pretty much any variable, but it will not help much for a new data set.

If you have 2 predictors that are correlated with each other then an interaction between the 2 will be fairly similar to a quadratic term on either, looking at only the data cannot tell you which is the most meaningful, you need to understand the nature of the process that generated the data to make these decisions.

Also your goals can influence things, are you trying to create a model to predict future outcomes? Are you trying to understand if certain variables may cause the response even after adjusting for others? or something else? which is the best model can be different for different goals.