2
$\begingroup$

I have a prediction model that performs a binary classification. The model takes 3 independent predictor parameters variables that can only be integers. I have calculated true positive rates (TPR) and false positive rates for each combination of three predictors. Each dot in the figure below represents TPR (y-axis) vs FPR (x-axis) of each combination.

True positive rate vs False positive rate for all possible combinations of the three parameters

Since there are more than 1 parameter, I can not apply a conventional ROC curve, however is there a way to represent success of a model with multiple parameters? Or would it still be reasonable to construct ROC for best and worst case scenarios, such as the figure below? best case scenario

Improve this question
$\endgroup$
4
  • $\begingroup$ I think you need to edit this to clarify. What do you mean by an integer parameter? Do you mean predictor variables which can only take on integer values? What do the dots represent? Why are there different numbers of dots in each column, or is that just overprinting? $\endgroup$
    – mdewey
    Jan 30, 2018 at 13:10
  • $\begingroup$ It is not overprinting, they are just happened as a consequence of multiple predictor parameters. I edited the rest as you suggested, I hope it is clearer. $\endgroup$
    – odoluca
    Jan 30, 2018 at 13:17
  • $\begingroup$ Im also confused as to how you drew your curve. An roc curve is parameterized by varying the classification threshold in your model. Is this what you did? Why does the x axis stop short of 1? $\endgroup$
    – Matthew Drury
    Jan 30, 2018 at 16:48
  • $\begingroup$ I only drew lines over best tpr vs fpr scores. :D I drew it in paint. it stop short of 1 cuz I didnt bother drawing anymore. $\endgroup$
    – odoluca
    Jan 30, 2018 at 21:25

2 Answers 2

Reset to default
1
$\begingroup$

The main problem with using multiple variables in the way you suggest is that it's undefined: let's say you have two variables, so your ROC would use 2 thresholds T_A and T_B. If at some threshold combination (e.g. T_A = 1 and T_B = 1) an observation is positive according to T_A but negative according to T_B, what is the prediction? Combine using logical AND? Logical OR? Round the average?

Let's say you decide to combine the variables using the logical AND method. You can now sample 2D threshold space and create a scatterplot-like ROC. But there are issues with this, read on.

A normal 1-variable ROC samples 1D theshold space. A consequence of this is that a positive at threshold T1 can not be a negative at a threshold T2, where T2 < T. I.e. the ROC is monotonously increasing.

The issue with sampling 2+D threshold space is the curve can go up, down, left or right from a given state, and you can't unambiguously connect the dots with a line. This ROC can have weird shapes that can't really be described by an auc value, and may have multiple optimal threshold combinations.

This is why it is more common to fit the variables with a (logistic?) model, that basically boils everything down into a single 'prediction probability' that you can treat like a 1D ROC variable.

Improve this answer
$\endgroup$
1
$\begingroup$

You are showing individual ROC curves for each predictor, but I assume you have a multivariate model (e.g., logistic regression). What you should be showing is the ROC curve from the final model, using the predicted probability values obtained after applying the model to validation data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.