ROC curve for comparing probability of default models

Question

I'm trying to compare two different probability of default models together by roc curve.I calculated the PD for 8 company by two different models.I know about the basic of roc curve and i can calculate fpr and tpr. but i don't know how to plot the roc for different threshold in r or Matlab. any help?

edit :apparently roc curve couldn't help with low samples like mine. any other approach for comparing models?

Answer 1

Answer to Question 1. In R, the library pROC can help you create the RoC curve as well as to calculate gini. The generic R code for the same is following:

#Store predicted model output in a vector pred_model

#Here Reg is supposed to be your model and data is supposed to be stored in the frame called df. Assume that actual default/ non default status is present in DefaultStatus column (it should be a dichotomous classification)

pred_model <- predict(Reg, newdata = df,type="response")

roc(df\\\$DefaultStatus,df\\\$pred_model,plot-True)

Answer to Question 2. A confusion matrix, that identifies how many defaults were you able to capture correctly (hit rate) and how many non defaults you classified as defaults (false positive rate) can help you quantify model efficacy in this case. A good model should have high hit rate and low false positive rate - Although the weight you would assign to each may vary depending on your risk aversion vs opportunity cost. For instance, if it's absolutely important for you not to miss any default, you will prefer a model with high hit rate, even if it leads to several non defaults getting misclassified as defaults. On the other hand, if you are more concerned about lost opportunity by not extending credit to a creditworthy customer, you would place more emphasis on lowering the false positive rate.

As you mentioned yourself, a sample of 8 is too small to check the efficacy of models.