For a binary classification task, I have (1) a fuzzy model (Mamdani) that outputs a number between 0 and 20 (after defuzzification) and (2) a logistic regression that outputs a number between 0 and 1. Even though both systems are based on very different theoretical basis (possibility theory vs probability theory), I was wondering if it is possible to compare their performance with something like ROC AUC. The final user will use the outputs of both models to assess the risk that a certain event presents. As such, both systems can have a threshold of acceptable risk and this means, AFAIK, that they can be compared in terms of ROC AUC.


The question is: how can I compare the performance of a fuzzy model and a probabilistic classifier, assuming that both have a continuous output? My first bet was to use ROC AUC because I'm interested into the ranking ability of each model. Also, I'm asking this because this kind of comparison is very unusual, AFAIK.

  • $\begingroup$ it is unclear what you are asking. and the question is too general... $\endgroup$
    – Haitao Du
    Apr 5 '18 at 17:31
  • $\begingroup$ In a binary classification setting, if you have a probabilistic classifier that has a continuous output between 0 and 1 (the probability), you want to know what is an acceptable threshold to consider that the input belongs to class 1. In general, you want the threshold that best separates each class. When comparing probabilistic classifiers with outputs between 0 and 1, you can use ROC AUC to do that. $\endgroup$
    – echo66
    Apr 5 '18 at 17:37
  • $\begingroup$ You're saying both "I was wondering if it is possible [...]" and then in the next sentence "[...] they can be compared in terms of ROC AUC". What are you actually asking? Can you phrase an question ending with a question mark? $\endgroup$
    – Calimo
    Apr 5 '18 at 18:48
  • 1
    $\begingroup$ By the way if the question is "can I use ROC AUC?" the answer is obviously yes, as you stated yourself in the last sentence of your statement. And also, ROC curve doesn't care if your output is a probability or not, I'm not sure where this common belief comes from. $\endgroup$
    – Calimo
    Apr 5 '18 at 18:50
  • $\begingroup$ @Calimo I updated the post. Now I see my mistake. Thanks :) $\endgroup$
    – echo66
    Apr 6 '18 at 10:48

Well if I'm understanding the problem correctly you can make a binary classification model which takes as input the outputs of the fuzzy model, and then compare it to your original logistic regression model.

These are my assumptions: Lets assume you have two systems regarding transaction fraud. Historically you already know if that transaction was indeed a fraud, i.e. $y\in\{0,1\}$. Vector $Y$ represents if each $y_t$ trasaction was a fraud or not, $Y=[y_0,y_1,..,y_t]$. If $X$ is the matrix containing the samples and its features then the logistic regression model will be something like,

\begin{equation} \hat{Y} = \hat{f}_1(X,\hat{\theta}_1). \end{equation}

where $\hat{f}_1$ is a sigmoid with a linear combination of features ($\hat{\theta}_1$ are the coefficients of the linear regression).

If you make a matrix with the outputs of the fuzzy model, let's call it $X_{fuzzyfeatures}$ you can train a new model,

\begin{equation} \hat{Y} = \hat{f}_2(X_{fuzzyfeatures},\hat{\theta}_2). \end{equation}

Now you can compare $f_1$ and $f_2$ models using ROC. $f_2$ can and should be nonlinear.

  • $\begingroup$ Thanks for the response! By "outputs of the fuzzy model" are you referring to the final output that we get after defuzzification or the outputs of the membership functions? I'm confused because you used X in the notation for f2 hat. (continue on the next response) $\endgroup$
    – echo66
    Apr 6 '18 at 10:36
  • $\begingroup$ Also, what you seem to be doing is applying a sigmoid transformation in the same fashion we would do with model calibration (check scikit learn pages on calibration with sigmoid). So, you are only put both models in the same range but they are still very different in terms of theoretical interpretations. Can't I just apply ROC AUC out of the box to both in a binary classification task? After all, I'm just interested in the ranking ability in terms of risk. $\endgroup$
    – echo66
    Apr 6 '18 at 10:36
  • 1
    $\begingroup$ When you use logistic loss as a loss function your classifier will be "calibrated" (logistic regression uses this as default loss, but you can also use it on different classifiers such as NN and GBT). You can apply ROC/AUC out of the box to a binary classification task, it all depends on what you want to see on that comparison, e.g. measure the prediction power about different features in relation to the target. (Yes I meant doing a new model so you can transform the $[0,..,20]$ range into a probability so you can use ROC/AUC) $\endgroup$
    – abriosi
    Apr 6 '18 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.