# $\chi^2$-test for good calibration of a rating system; degrees of freedom

Assume that I have a rating system with $R$ rating classes, the classes have a probability of default $p_{i0}, i=1,2..,R$. The probability of default is the probability that a company in that class has a payment problem within one year.

I have a set of companies that I have to partition over these classes, using one or other model.

Let's say that I classified $N$ companies last year and each rating class contains $N_i$ companies where $\sum_{i=1}^R N_i=N$. Today, one year later, I count the observed number of defaults in each class and I find $d_i$.

I want to test whether my system is well calibrated, i.e. whether these observed numbers of defaults are in line with my a priori fixed probabilities $p_{i0}$.

I define my test statistic as $X^2=\sum_{I=1}^R \frac{(d_i-N_ip_{i0})^2}{N_i p_{i0}}$. If the expected cell counts $N_ip_{i0}$ are not too low and if the defaults in different rating classes are independent, then this test statistic is $\chi^2$, but what is the number of degrees of freedom , is it $R$ (because I have a sum of $R$ squared standard normal random variables that are independent) or is it $R-1$ as in pearson's $\chi^2$ ? If the latter is the case; where do I loose one degree of freedom ( a reference is fine also).

• Wouldn't it be easier to simulate it? You have a model of the data generating process (DGP). You have what you think is a realization of this DGP. Use the probabilities that were fixed a priori to draw a random sample of the same size as the one that you're interested. Compute the statistic. And repeat this a 1000,000 times. Is your statistic measuring the discrepancy between the model and the realization larger or smaller than the threshold value beyond which you say that it isn't well-calibrated? – Toby May 23 '17 at 12:11
• @Toby could you elaborate on that in an answer? – user83346 May 24 '17 at 15:05
• I believe my account of the chi-squared test at stats.stackexchange.com/a/17148/919 completely answers your question. It also provides a standard reference (Kendall & Stuart). – whuber May 24 '17 at 18:14
• @whuber: It is a very good answer and well written, but I don't see in that answer whether it is $R$ or $R-1$ degrees of freedom ? – user83346 May 25 '17 at 8:34
• If I may be so bold as to quote from my own post, and substitute your numbers and variable names for the names used there, it reads "I know in advance that the sum of the counts must equal $N$. That's one relationship. I estimated zero parameters from the data. That's zero additional relationships, giving $0+1=1$ total relationships. Presuming they (the parameters) are all (functionally) independent, that leaves only $R−0−1 =R-1$ (functionally) independent 'degrees of freedom': that's the value to use for $\nu$." It remains to you to verify the conditions I enumerated after that passage. – whuber May 25 '17 at 21:38

This is an elaboration on the comment made earlier.

If I've understood the problem correctly then you have a model and you want to know whether this model is a good model of reality. The model you have is a model of company defaults. There are $R-$different kinds or classes of companies. Each class consists of $N_i-$companies. And each company in these $R-$classes has a probability of $p_i$ of defaulting in a year. You know the number of companies in each class and you have a guess as to what $p_i$ is. And what you ask when you ask whether it's a good model is whether these probabilities are good guesses.

Now to measure whether this guess as to what the probabilities are good you quantify the discrepancy between the model and reality by taking the squared difference between what was expected under the model and what actually happened divided by what was expected under the model. This statistic follows a $\chi^2-$distribution. You don't know however which one.

What I am suggesting is to treat your model as the null hypothesis. Assume that the model is true and that for some reason you've managed to guess the right probabilities. How would the distribution of this statistic look like if the model were true? To find out you:

1. Run $R-$ binomial trials of size $N_i$ with probability of success $p_i$ with $i=1,2,..,R$. These are your classes and defaults;

2. Divide the number of successes by the number of trials for each class separately. This is $N_i p_i$ in the statistic.

3. Compute the statistic by comparing this simulation to the model and store it;

4. Repeat steps 1 - 3 a large number of times e.g. 1,000,000;

5. See what proportion of these simulated statistics are larger than the statistic you've computed by comparing your model to reality. If this proportion is smaller than say 5% or some other threshold value, then you have a statistically significant result.

On a side note: why are you using this statistic? Why not quantify how large the losses are from acting on this model as if it were true or how much money is lost through defaults and do steps 1-5 for that. This would seem more meaningful to me as this is what you're trying to avoid, correct?