2
$\begingroup$

Although the choice of machine learning algorithms is more of an art than a science, a few common guidelines have been compiled on the matter, namely this diagram from scikit-learn as well as this comparison table.

However, the more I stare at those guidelines, the more I am confused as to the rationale behind them. Is there a comprehensive document that explains the why behind model selection? For example, why does the scikit-learn diagram state that an SGD classifier is better than a linear SVC when one has more than 100K samples, etc.?

$\endgroup$
  • $\begingroup$ Been wondering this myself recently, considering making a similar post—I can't seem to find a comprehensive guideline with references. Perhaps we could put one together as a CV community? I often think, "I don't even kno where to start" when I call names(caret::getModelInfo()) $\endgroup$ – Mark White Jul 1 '17 at 19:00
  • $\begingroup$ Those diagrams give me the analyst's creeps. Their potential of misuse is stupendous. While not glaringly wrong, if one needs such a diagram, one needs to study ML/Stats more in general. (Eg. The fact that EDA is not at all mentioned in that diagram makes it inapplicable for any real-life task.) $\endgroup$ – usεr11852 Jul 1 '17 at 23:30
  • $\begingroup$ I think "SGD classifier vs. linear SVC" is more an algorithm selection (in scikit-learn): The underlying model is the same, but the SGD fitting (training) computation is scalable to larger data sets. $\endgroup$ – GeoMatt22 Jul 1 '17 at 23:32
  • $\begingroup$ @GeoMatt22 The whole question in this case is precisely why: Why is the SGD better scalable? What is the conceptual motivation for the diagram to branch off the way it does? $\endgroup$ – Tfovid Jul 2 '17 at 16:45
  • $\begingroup$ From here: "SGDClassifier can optimize the same cost function as LinearSVC by adjusting the penalty and loss parameters. In addition it requires less memory, allows incremental (online) learning, and implements various loss functions and regularization regimes." I bolded the part that I think is the motivation. $\endgroup$ – GeoMatt22 Jul 2 '17 at 20:29
1
$\begingroup$

Note: I'm not an expert (here to learn - corrections are welcome).

Although the choice of machine learning algorithms is more of an art than a science

For supervised learning problems, an empirical (as opposed to artistical) approach to model selection involves assessing a model's goodness of fit using some performance criterion. In other words, the accuracy of a chosen model can be objectively evaluated by using a chosen metric as a performance measure. Choice of performance criterion is informed by the nature of the problem (i.e. classification vs. prediction). Concrete examples of this can be found in competitions on kaggle:

  • If there are N images, you will be making 17N predictions. Submissions are scored on the log loss: $$-\frac{1}{N}\sum_{i=1}^{N}[y_{i} log(\hat y_{i})+(1-y_{i})log(1-\hat y_{i})]$$ where:

    • N is the 17 * the number of scans in the test set
    • $\hat y_{i}$ is the predicted probability of the scan having a threat in the given body zone
    • $y_{i}$ is 1 if a threat is present, 0 otherwise
    • $log()$ is the natural (base e) logarithm1
  • Submissions are evaluated on Mean Absolute Error between the predicted log error and the actual log error. The log error is defined as $$logerror = log(Zestimate) - log(SalePrice)$$ and it is recorded in the transactions training data. If a transaction didn't happen for a property during that period of time, that row is ignored and not counted in the calculation of MAE.2

  • Submissions will be evaluated based on their mean F1 score.3

  • Submissions are evaluated on the $R^{2}$ value, also called the coefficient of determination.4

A more general example of this is error minimization: the selected performance criterion is an error value such as mean squared error (MSE), and model performance is evaluated based on MSE minimization, where the model with the lowest MSE is determined to be the best estimate of true but unknown function $f$.

I suppose one could think of model selection as a kind of optimization problem, in which one optimizes for the chosen measure of performance.

Is there a comprehensive document that explains the why behind model selection?

Given the number of different modeling methods and model selection criteria, I would be surprised if a single documents provides comprehensive coverage of model selection.


1. Passenger Screening Algorithm Challenge - Evaluation

2. Zillow Prize: Zillow’s Home Value Prediction (Zestimate) - Evaluation

3. Instacart Market Basket Analysis - Evaluation

4. Mercedes-Benz Greener Manufacturing - Evaluation

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I don't think this question is about model selection in the conventional sense of the term, but about how to decide what models to consider in the first place. Only once you've chosen some models to consider can you bring formal model-selection or model-evaluation methods to bear like Bayes factors or AIC or the comparison of cross-validated predictive accuracy. $\endgroup$ – Kodiologist Jul 1 '17 at 22:32
  • $\begingroup$ @Kodiologist Oh I see - I guess I missed the real point of the question. Is it ok if I leave this answer, or should I just delete it? $\endgroup$ – julian Jul 1 '17 at 22:50
  • 1
    $\begingroup$ ¯\_(ツ)_/¯ You don't have to delete it; it's not grossly inappropriate or anything. It's your call. $\endgroup$ – Kodiologist Jul 1 '17 at 22:53
  • $\begingroup$ @Kodiologist understood. I guess I will leave it for now, just in case another novice draws incorrect conclusions about what is really being asked $\endgroup$ – julian Jul 1 '17 at 23:00
  • $\begingroup$ I upvoted both of Kodio's comments; he is right but this is a nice effort (+1). $\endgroup$ – usεr11852 Jul 1 '17 at 23:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.