Evaluation and optimization in machine learning I am reading the article "A few useful things to know about machine learning" by P. Domingos. Now, I am confused about two of the three components of learning, i.e., the evaluation and optimization. From the article, the evaluation function is the one used to distinguish good classifiers from bad ones, while the optimization is the method to search among all the classifiers for the highest-scoring one. So, they are kind of similar, right? Am I misunderstanding something here? Many thanks for your time and attention.
 A: Evaluation is a score/metric that tells us how we are doing. For example, if we are trying to predict a number across a test set of size $n$, we can compute Mean Absolute Error = $\frac{1}{n}\sum_{i=1}^{n}|observed_i - predicted_i|$ or we can compute Root Mean Squared Error = $\sqrt{\frac{1}{n}\sum_{i=1}^{n}(observed_i - predicted_i)^2}$
Optimization refers to the representation and how we can choose different ways to optimize it. For example, we can try to simply try every possible hypothesis in our hypothesis space. Or we can use a more intelligent method to try the most promising hypothesis only. And as we are optimizing, we use our evaluation function to know if this particular hypotheses is better than another.
A: Here is a very clean answer.
Optimization consists of minimizing or maximizing an objective function with respect to one or more variables (a.k.a. parameters, a.k.a. arguments), possibly subject to constraints on those variables.
Viewed in this framework, "evaluation" means evaluating the objective function at a particular combination of variable values.  The optimizer has to evaluate the objective function (and possibly its gradient and maybe Hessian) at various values of the variables in order to carry out the optimization.
