Rationale for model selection in machine learning Although the choice of machine learning algorithms is more of an art than a science, a few common guidelines have been compiled on the matter, namely
this diagram from scikit-learn as well as this comparison table.
However, the more I stare at those guidelines, the more I am confused as to the rationale behind them. Is there a comprehensive document that explains the why behind model selection? For example, why does the scikit-learn diagram state that an SGD classifier is better than a linear SVC when one has more than 100K samples, etc.?
 A: Note: I'm not an expert (here to learn - corrections are welcome).

Although the choice of machine learning algorithms is more of an art than a science

For supervised learning problems, an empirical (as opposed to artistical) approach to model selection involves assessing a model's goodness of fit using some performance criterion. In other words, the accuracy of a chosen model can be objectively evaluated by using a chosen metric as a performance measure. Choice of performance criterion is informed by the nature of the problem (i.e. classification vs. prediction). Concrete examples of this can be found in competitions on kaggle:


*

*
If there are N images, you will be making 17N predictions. Submissions are scored on the log loss:
  $$-\frac{1}{N}\sum_{i=1}^{N}[y_{i} log(\hat y_{i})+(1-y_{i})log(1-\hat y_{i})]$$
  where:
  
  
*
  
*N is the 17 * the number of scans in the test set
  
*$\hat y_{i}$ is the predicted probability of the scan having a threat in the given body zone
  
*$y_{i}$ is 1 if a threat is present, 0 otherwise
  
*$log()$ is the natural (base e) logarithm1


*
Submissions are evaluated on Mean Absolute Error between the predicted log error and the actual log error. The log error is defined as
  $$logerror = log(Zestimate) - log(SalePrice)$$
  and it is recorded in the transactions training data. If a transaction didn't happen for a property during that period of time, that row is ignored and not counted in the calculation of MAE.2


*
Submissions will be evaluated based on their mean F1 score.3


*
Submissions are evaluated on the $R^{2}$ value, also called the coefficient of determination.4

A more general example of this is error minimization: the selected performance criterion is an error value such as mean squared error (MSE), and model performance is evaluated based on MSE minimization, where the model with the lowest MSE is determined to be the best estimate of true but unknown function $f$. 
I suppose one could think of model selection as a kind of optimization problem, in which one optimizes for the chosen measure of performance.

Is there a comprehensive document that explains the why behind model selection?

Given the number of different modeling methods and model selection criteria, I would be surprised if a single documents provides comprehensive coverage of model selection. 


*

*I found chapter 2 of An Introduction to Statistical Learning to be a good starting point for understanding the general approach, but this is just my opinion. Chapter 5  "Resampling Methods" discusses cross-validation and other techniques, Chapter 6 "Linear Model Selection and Regularization"

*Chapter 7 of The Elements of 
Statistical Learning, "Model Assessment and Selection"

*Chapter 5 of Modern Multivariate Statistical Techniques, "Model Assessment and Selection in Multiple Regression"

*Chapter 4, chapter 5, chapter 11 and chapter 20 of Applied Predictive Modeling discuss model performance evaluation in various contexts



1. Passenger Screening Algorithm Challenge - Evaluation
2. Zillow Prize: Zillow’s Home Value Prediction (Zestimate) - Evaluation
3. Instacart Market Basket Analysis - Evaluation
4. Mercedes-Benz Greener Manufacturing - Evaluation
