3
$\begingroup$

I am trying to work on a project for school and I am currently trying to select the best model (models) for the classification task. I have a few questions regarding this:

  1. What does a baseline model mean and how can I pick/select one.
  2. Do I need to select a baseline even though I am going to compare several algorithms and conduct hyperparameter tuning later using cross-validation?
  3. If it is necessary Does it make sense to use nested cross-validation just for the sake of choosing a baseline/ comparing different algorithms with the default parameters? just like in the code below?
# Split-out test dataset
X = features_ds
y = labels_ds
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, shuffle=True)

# Model selection 
#evaluation - baselines
models = []
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVC', SVC()))
models.append(('AdaBoost', AdaBoostClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('GBClassifier', GradientBoostingClassifier()))
models.append(('lightgbm', LGBMClassifier()))
models.append(('XGBoost', XGBClassifier()))
models.append(('ExtraTreesClassifier', ExtraTreesClassifier()))

# evaluate each model in turn (spot checking algorithms (of baselines))
results = []
names = []
seed=7
num_folds=10
for name, model in models:
   kfold = StratifiedKFold(n_splits=num_folds, random_state=seed, shuffle=True)
   cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring='accuracy')
   results.append(cv_results)
   names.append(name)
   print('%s: %f (%f)' % (name, cv_results.mean(), cv_results.std()))

# Compare Algorithms
pyplot.boxplot(results, labels=names)

```
$\endgroup$
1
$\begingroup$
  1. A baseline model is any model you like that you want to compare your performance against after going through some kinds of optimisations. So, there is no principled way of choosing it. But, in general, simple models are preferred.

  2. You don't need to. It's just a good practice to see how much you get if you use more complex models. For example, if a simple linear regression achieves negligibly (due to the business problem) worse performance than a deep neural net, sometimes you'd choose the regression due to some computational concerns.

  3. There is only one cv loop in your code, so it's not nested. But, it makes sense to go over each baseline candidate and compare their cross validated performances.

$\endgroup$
1
$\begingroup$

Consider the following paragraph from Data Science for Business by Foster Provost and Tom Fawcett.

In some cases just demonstrating that a model generates some (nonzero) profit, or a positive return on investment, will be informative by itself. Nevertheless, another fundamental notion in data science is: it is important to consider carefully what would be a reasonable baseline against which to compare model performance. This is important for the data science team in order to understand whether they indeed are improving performance, and is equally important for demonstrating to stakeholders that mining the data has added value.

A baseline classifier is meant to be a simple, reasonable classifier which you can use as a baseline for evaluating more complex classifiers. An example of a baseline classifier is to always predict the most predominant class. If 70% of animals in your training set are dogs, you can predict "dog" every time and have 70% accuracy. This is called the majority classifier and any model you build should at least be better than it. Another example is a decision tree with only one split (also called a decision stump). The ideal baseline classifier depends on the situation and domain knowledge. Consider the following excerpt from The Signal and the Noise, reproduced in Data Science for Business, which gives an example of using domain knowledge of weather forecasting to select two appropriate baseline models.

There are two basic tests that any weather forecast must pass to demonstrate its merit: It must do better than what meteorologists call persistence: the assumption that the weather will be the same tomorrow (and the next day) as it was today. It must also beat climatology, the long-term historical average of conditions on a particular date in a particular area.

So, to answer your questions directly:

  1. Pick a baseline classifier based on your knowledge of the domain, or else choose some sort of simple classifier which you know will give an okay result.

  2. Yes, the baseline classifier is used for comparing/interpreting the performance of your other models.

  3. If you choose a baseline model with hyperparameters then you should tune them, but in general, your baseline model should not be chosen from a large set of models in order to maximize performance and you should know that it will give a reasonable result before implementing it.

$\endgroup$
3
  • $\begingroup$ When you say "Pick a baseline classifier based on your knowledge of the domain, or else choose some sort of simple classifier which you know will give an okay result." Do you mean by looking in literature and see what models they have been using and what worked for them? e.g. for crop type classification, I noticed (from papers) that RF, SVM, and CNN are the most used and the ones that achieved the highest performance (when compared to other models). So, in this case, should I use RF and/or SVM as my baselines (without hyperparameter tuning)? isn't RF and SVM called benchmarks? (in this case). $\endgroup$ – Rim Sleimi Jun 3 '20 at 17:33
  • $\begingroup$ It depends on what you're trying to do. Often, you compare against a baseline to justify a model which is more complicated or represents a novel approach. For your case, perhaps choose the simplest or most classical model as the baseline. You do this because, all other things being equal, that's the one you should probably default to. Your goal will be to show that you can beat that model using other more complicated or novel techniques and that justifies using the new model. If all models are equally simple/valid and you don't propose anything new, you may not need a baseline. $\endgroup$ – Ryan Volpi Jun 3 '20 at 17:51
  • $\begingroup$ Thanks for your response ! $\endgroup$ – Rim Sleimi Jun 3 '20 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.