# choosing a baseline algorithm using a nested cross validation

I am trying to work on a project for school and I am currently trying to select the best model (models) for the classification task. I have a few questions regarding this:

1. What does a baseline model mean and how can I pick/select one.
2. Do I need to select a baseline even though I am going to compare several algorithms and conduct hyperparameter tuning later using cross-validation?
3. If it is necessary Does it make sense to use nested cross-validation just for the sake of choosing a baseline/ comparing different algorithms with the default parameters? just like in the code below?
# Split-out test dataset
X = features_ds
y = labels_ds
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, shuffle=True)

# Model selection
#evaluation - baselines
models = []
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVC', SVC()))
models.append(('RF', RandomForestClassifier()))
models.append(('lightgbm', LGBMClassifier()))
models.append(('XGBoost', XGBClassifier()))
models.append(('ExtraTreesClassifier', ExtraTreesClassifier()))

# evaluate each model in turn (spot checking algorithms (of baselines))
results = []
names = []
seed=7
num_folds=10
for name, model in models:
kfold = StratifiedKFold(n_splits=num_folds, random_state=seed, shuffle=True)
cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring='accuracy')
results.append(cv_results)
names.append(name)
print('%s: %f (%f)' % (name, cv_results.mean(), cv_results.std()))

# Compare Algorithms
pyplot.boxplot(results, labels=names)

$$$$
`

1. A baseline model is any model you like that you want to compare your performance against after going through some kinds of optimisations. So, there is no principled way of choosing it. But, in general, simple models are preferred.

2. You don't need to. It's just a good practice to see how much you get if you use more complex models. For example, if a simple linear regression achieves negligibly (due to the business problem) worse performance than a deep neural net, sometimes you'd choose the regression due to some computational concerns.

3. There is only one cv loop in your code, so it's not nested. But, it makes sense to go over each baseline candidate and compare their cross validated performances.

Consider the following paragraph from Data Science for Business by Foster Provost and Tom Fawcett.

In some cases just demonstrating that a model generates some (nonzero) profit, or a positive return on investment, will be informative by itself. Nevertheless, another fundamental notion in data science is: it is important to consider carefully what would be a reasonable baseline against which to compare model performance. This is important for the data science team in order to understand whether they indeed are improving performance, and is equally important for demonstrating to stakeholders that mining the data has added value.

A baseline classifier is meant to be a simple, reasonable classifier which you can use as a baseline for evaluating more complex classifiers. An example of a baseline classifier is to always predict the most predominant class. If 70% of animals in your training set are dogs, you can predict "dog" every time and have 70% accuracy. This is called the majority classifier and any model you build should at least be better than it. Another example is a decision tree with only one split (also called a decision stump). The ideal baseline classifier depends on the situation and domain knowledge. Consider the following excerpt from The Signal and the Noise, reproduced in Data Science for Business, which gives an example of using domain knowledge of weather forecasting to select two appropriate baseline models.

There are two basic tests that any weather forecast must pass to demonstrate its merit: It must do better than what meteorologists call persistence: the assumption that the weather will be the same tomorrow (and the next day) as it was today. It must also beat climatology, the long-term historical average of conditions on a particular date in a particular area.