How do we usually select the best combination of parameters of a machine learning model (for a given dataset)? Am I wrong, or the standard way of optimizing a machine learning model is by evaluating the algorithm over the (initial) dataset for all possible combinations of parameters, and then pick up the one that gives the 'best' evaluation score? If we still don't get an acceptable score, we should then slice and dice the (initial) dataset and start the process from the beginning, but this time applying it on the new dataset? Does this sounds right to you or I am missing something here?
 A: You are missing some things. If you have a very complicated set of models, choosing the one which agrees with the training data most closely will overfit. For example, if you memorize the training data, you will often generalize poorly. There are many techniques for avoiding or reducing overfitting, such as cross-validation (that it is the name of the site suggests it is something important), but they are not described in your procedure. 
It is often difficult, or even intractible, to find the parameters which give the best model. For example, training techniques for neural networks almost always find suboptimal local minimums. You can't just jump to the global optimum, but some local minimums are better than others. You are glossing over this issue.
I'm not sure what you mean by "slice and dice the dataset." Sometimes you need new data, more examples or a different type of data. Sometimes you need to clean up the old data. Sometimes you need to use a different set of models. I wouldn't call these slicing and dicing.
