2
$\begingroup$

I recently came up with the following structure to assess which models work best for some simple classification tasks. Assume that we have some labelled multivariate data, already splitted into X and y.

I am interested, in whether the following structure is decent or how I could improve. Any feedback is appreciated. Thanks!

N = 10
model = sklearn.discriminant_analysis.LinearDiscriminantAnalysis()  # choose any model, e.g. LDA

for i in range(N):
    
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
    
    # Scaling (or Transforming alternatively)
    scaler = StandardScaler()
    scaler.fit(X_train)
    X_train[:] = scaler.transform(X_train)
    X_test[:] = scaler.transform(X_test)
    
    model.fit(X_train, y_train)
    
    accIS = model.score(X_train, y_train)
    accOS = model.score(X_test, y_test)
    
    train_list.append(accIS)
    test_list.append(accOS)
    
    print(i) # keep track of iterations, e.g. for ANNs

print(np.mean(train_list), np.mean(test_list))

y_pred_train = model.predict(X_train)
print("In-sample CM: ", confusion_matrix(y_train, y_pred_train))
    
y_pred_test = model.predict(X_test)
print("Out-sample CM: ", confusion_matrix(y_test, y_pred_test))

The process can be described as follows:

Do N-times:

  • split the data randomly in training, test samples
  • apply e.g. a scaler or transformer function to data, if necessary
  • fit the model on the training data
  • calculate the accuracy (or some other metric) of the training and test data
  • store training and test accuracies in separate lists

Finally take the mean accuracy of each of the two lists in order to see how the model performed on average on training and test data.

$\endgroup$
9
  • 3
    $\begingroup$ For people who do not know or do not want to read Python code, what is your process? $\endgroup$
    – Dave
    Apr 2 at 15:23
  • 2
    $\begingroup$ Why test one model type only? Also, I understand that you want to do the iterations to kind of check how robust your modeling is. Why not do k-fold cross-validation? In this case, you make sure that every sample has been tested once and trained on k-1 times. Also, maybe worth checking, in MATLAB there is a nice fitcauto function that tests a variety of models for you: de.mathworks.com/help/stats/fitcauto.html. Scikit-learn still did not implement this, but you can do it yourself. $\endgroup$
    – Tino D
    Apr 2 at 15:52
  • 1
    $\begingroup$ @TinoD: your comment looks like an answer, want to post it as such? $\endgroup$ Apr 2 at 16:24
  • 2
    $\begingroup$ Thanks for the hint, I'll do so tomorrow morning, i'll put a bit more effort in an answer than just what I wrote in the comment @StephanKolassa $\endgroup$
    – Tino D
    Apr 2 at 16:26
  • 1
    $\begingroup$ @TinoD: Better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$ Apr 2 at 16:30

1 Answer 1

3
$\begingroup$

As mentioned in my comment, your approach actually somehow mimics what k-fold cross validation does: you want to train and test on different subsets of the data. However, you will find that for low values of N in your code, the metrics will change significantly for each time you are training and testing on those different subsets. Especially if your dataset has, among other things, outliers or imbalanced classes.

To define a cross validation partition in scikit-learn, use the following tutorial.

Finally, I highly recommend not testing on one model type only. Try two or three (different) models and do a sanity check at the end:

  1. why is my model good/ bad?
  2. how does my confusion matrix look like?
  3. (for binary problems) how are my F1-score and recall looking like

I will try to make time to do an example in python tomorrow.

Edit: Example in python with k-fold cross-validation

In the below code, you will find a classification example on the iris dataset using linear SVM. I added as many comments as I can:

from sklearn.datasets import load_iris
from sklearn.model_selection import KFold
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
iris = load_iris() # get the iris dataset
predictors = iris.data # , the predictors
response = iris.target # and the response (classes)
k = 5 # specify the kfold, typically 5, but there are also ways to determine an optimal k
kf = KFold(n_splits=k, shuffle=True) # define the partition, you can also specify a seed for reproducibility
# to define a parition on imbalanced data, check StratifiedKFold from the same library
validationAccuracy = [] # define a list to store the validation accuracy scores
trainingAccuracy = [] # , training scores
models = [] # and for the models
# you can also store the predictions for plotting...
for foldIdx, (trainIdx, testIdx) in enumerate(kf.split(predictors), 1): # for each k
    # splitting
    xTrain, xTest = predictors[trainIdx], predictors[testIdx] # get the training data and response
    yTrain, yTest = response[trainIdx], response[testIdx] # and the testing (validation) data and response
    # training
    clf = SVC(kernel='linear') # define
    clf.fit(xTrain, yTrain) # and fit the model to the training data
    # predicting
    yPredValidation = clf.predict(xTest) # get the predictions for the test partition
    yPredTraining = clf.predict(xTrain) # get the predictions for the training partition
    # testing
    valAcc = accuracy_score(yTest, yPredValidation) # get accuracy of validation
    traAcc = accuracy_score(yTrain, yPredTraining) # and the accuracy of training
    # storing
    validationAccuracy.append(valAcc) # store validation accuracy
    trainingAccuracy.append(traAcc) # , training accuracy
    models.append(clf) # and model
    ###print out summaries###
    print(f"Fold {foldIdx}: Training Accuracy = {traAcc:.2f}, Validation Accuracy = {valAcc:.2f}")

As mentioned, this is a better way to approximate the performance of the modeling on different partitions of the data, since it is thorough. K-fold partitions make sure that every record in your dataset is tested on. Also, just looking at accuracies is not enough. Looking at the confusion matrix is essential as well.

To know how certain the predictions are, you need to look at the probabilities of the predictions.

$\endgroup$
2
  • 3
    $\begingroup$ +1, thank you! (I do have reservations about F1 and recall, though, which suffer from the same issues as accuracy.) $\endgroup$ Apr 3 at 6:23
  • 1
    $\begingroup$ Agree with you! The notion of discrete classifications without looking at or even understanding the underlying probabilities is not optimal and usually leads to uncovering many flaws during implementation $\endgroup$
    – Tino D
    Apr 3 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.