Let's imagine I have two completely different multi-class ML models, let's call them ML1 and ML2. The models were trained on completely different data with different target classes. As an output, the models print the distributions over the target classes. For example, something like this

ML1: [0.1, 0.2, 0.7]
ML2: [0.1, 0.1, 0.3, 0.4, 0.1]

According to the output, ML1 decides that it's input instance belongs to the 3rd class and ML2 decides that it's input instance belongs to the 4rd.

What I want to do is to decide which model is surer about it's prediction. Apparently I cannot directly compare the models output. Somehow I should normalize the outputs. I would appreciate any help.

  • $\begingroup$ This question needs more context on the data you are using. Are the data sets used by ML1 and ML2 refering to the same kinds of entities behind the data but merely the class distinction is different (e.g. the data used for both models refers to trucks, but the class variable of one simply represents a more granular distinction) or are these data sets entirely different? $\endgroup$ – deemel Feb 8 '19 at 10:05
  • $\begingroup$ @Rickyfox, Thank you very for your comment! Regarding the question, the models are completely different and the data is completely different. $\endgroup$ – com Feb 8 '19 at 11:48
  • $\begingroup$ What meaninful comparison do you try to make between different models that solve a different problem based on different features ? $\endgroup$ – deemel Feb 8 '19 at 12:06
  • $\begingroup$ @Rickyfox, What I want to do is to identify what model is surer about it's prediction. Apparently, the direct comparison is not a solution. $\endgroup$ – com Feb 8 '19 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.