6
$\begingroup$

I have applied various ML models (fundamental and ensemble) to the same dataset for classification problem solving.

AdaBoost, Bagging, and XGBoost classifiers gave the best accuracies. However, they vary from 90 to 94%. I have done 5-fold cross validation.

Other plain old basic ML algorithms gave me 60-80%.

RNN with attention gave me 83%.

How can I explain this difference to my supervisor? I mean, why are they giving different accuracies?

$\endgroup$
7
  • $\begingroup$ One reason is different models have different capacities (stats.stackexchange.com/questions/312424/…). But you also have to ensure you're evaluating your model correctly, i.e. you've perform the train/dev/test split correctly - XGBoost can easily over-fit your training data but often perform poorly on new/unseen data. If the test split isn't representative of unseen data, the high accuracy won't be seen in practice and the "other" algorithms may perform as well (or better). I'd also look at "regularization" $\endgroup$ Commented Jul 11 at 5:20
  • $\begingroup$ @DavidWaterworth, I have done 5-fold cross validation. $\endgroup$
    – user366312
    Commented Jul 11 at 5:57
  • $\begingroup$ Yeah that's often problematic if say you have 5 samples that are "related in some way" and 1 of each ends up in each fold. Plus you should consider how trees work, it's possible given suitable hyper-parameters to get 100% training score for any model - i.e. a tree can create a "box" around every training example and assign it the correct label (i.e. overfitting). But these "boxes" can have somewhat arbitrary boundaries. Some of the other methods have smoother boundaries so less likely to overfit. $\endgroup$ Commented Jul 11 at 7:27
  • 1
    $\begingroup$ @user366312 you used five-fold cross-validation for performance evaluation? How did you perform model selection (also referred to as "hyperparameter tuning"). Most modern learning methods are heavily dependent on hyperparameter tuning, so if you don't use the correct tuning method for each classifier system, it is easy to introduce substantial biases into the performance evaluation. $\endgroup$ Commented Jul 11 at 14:32
  • $\begingroup$ Related? $\endgroup$
    – Dave
    Commented Jul 11 at 14:45

3 Answers 3

7
$\begingroup$

The better performing models have a better match between the model and the problem. That's about all there is to it. The No Free Lunch theorem tells us that when averaged over all possible problems, no algorithm is actually any better than another - it is not possible to find a method which will produce strictly superior results to another in all settings. Here, you have a good fit between a method and a problem, and find that some particular methods perform better on problems of this sort. This isn't really surprising, it's common that different methods will perform differently on the same problem. Had you applied these same methods to a different problem, though, you may well have gotten different results - there are problems where the very basic ML models would indeed outperform more complex ones. It's a bit of a fallacy to ask "why is this ML model better", as none actually are - the right question to ask is "why is this ML model better in this particular context".

If you want a better understanding of "what the model is doing", you could perhaps have a look at the features used by each model and their relative importance or model weights to gain a qualitative understanding of how the different models make their predictions. This may be reasonable or tractable with a simple model with few parameters, but a highly complex model with many parameters is often referred to as a "black box" because of our inability to qualitatively understand what's going on inside the model. Complex models may be good at mapping inputs to the desired outputs, but it may be difficult to explain the "how" or "why" of their operation.

$\endgroup$
5
$\begingroup$

All models involve, to an extent, finding a "sweet spot" in terms of parsimony and complexity to ensure generalizable predictions across a range of possible predictor values. Some models are more or less flexible, in this regard, to be tuned with a given level of complexity to achieve that purpose.

So why do we see variability in model results? In part you are begging the question. If I fit twenty or more models, I still expect a range of performances, some may show good predictiveness, some less so. If all models provided the same result in all settings, then why bother developing the theory around them individually? On the other hand, a demonstrably inferior model might still provide compelling and usable results.

For a given panel of ML models to be compared/combined, there is a background distribution of performances which does not depend on the data. You might just simulate some toy data to demonstrate this before moving forward - you can spare your advisor's time by discarding models that aren't expected to give competitive results.

As for your specific findings, boosting and bagging go hand-in-glove, they are operationally and practically similar. While we can't be sure what your "other algorithms" (aside from RNN) are, one can compare it to noting that logistic and probit regression results are very similar and better than linear regression for modeling a binary endpoint.

$\endgroup$
3
  • $\begingroup$ Other algorithms are plain old basic ML algorithms. $\endgroup$
    – user366312
    Commented Jul 11 at 6:27
  • $\begingroup$ I am solving a classification problem. $\endgroup$
    – user366312
    Commented Jul 11 at 6:28
  • 2
    $\begingroup$ @user366312 I do not believe there is any such thing as a "plain old basic ML algorithm". $\endgroup$
    – AdamO
    Commented Jul 11 at 13:57
4
$\begingroup$

First off, accuracy is problematic as a KPI: Why is accuracy not the best measure for assessing classification models?

However, the same "effect" will happen also with other KPIs.

And the short answer is "different models give different results because they are different models". If they gave the same result, they would not be different models any more. One model may be too rigid in its treatment of predictors. Another one may be too flexible and overfit. One model may work well in one part of the predictor-outcome space, another one in a different part, and now it comes down to which part of this space your data mostly lie in.

There is no reason to expect equal or even only approximately equal results from different models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.