$\chi^2$-test for good calibration of a rating system; degrees of freedom Assume that I have a rating system with $R$ rating classes, the classes have a probability of default $p_{i0}, i=1,2..,R$.  The probability of default is the probability that a company in that class has a payment problem within one year. 
I have a set of companies that I have to partition over these classes, using one or other model.  
Let's say that I classified $N$ companies last year and each rating class contains $N_i$ companies where $\sum_{i=1}^R N_i=N$. Today, one year later, I count the observed number of defaults in each class and I find $d_i$. 
I want to test whether my system is well calibrated, i.e. whether these observed numbers of defaults are in line with my a priori fixed probabilities $p_{i0}$. 
I define my test statistic as $X^2=\sum_{I=1}^R \frac{(d_i-N_ip_{i0})^2}{N_i p_{i0}}$.  If the expected cell counts $N_ip_{i0}$ are not too low and if the defaults in different rating classes are independent, then this test statistic is $\chi^2$, but what is the number of degrees of freedom , is it $R$ (because I have a sum of $R$ squared standard normal random variables that are independent) or is it $R-1$ as in pearson's $\chi^2$ ? If the latter is the case; where do I loose one degree of freedom ( a reference is fine also).
 A: This is an elaboration on the comment made earlier.
If I've understood the problem correctly then you have a model and you want to know whether this model is a good model of reality. The model you have is a model of company defaults. There are $R-$different kinds or classes of companies. Each class consists of $N_i-$companies. And each company in these $R-$classes has a probability of $p_i$ of defaulting in a year. You know the number of companies in each class and you have a guess as to what $p_i$ is. And what you ask when you ask whether it's a good model is whether these probabilities are good guesses. 
Now to measure whether this guess as to what the probabilities are good you quantify the discrepancy between the model and reality by taking the squared difference between what was expected under the model and what actually happened divided by what was expected under the model. This statistic follows a $\chi^2-$distribution. You don't know however which one.
What I am suggesting is to treat your model as the null hypothesis. Assume that the model is true and that for some reason you've managed to guess the right probabilities. How would the distribution of this statistic look like if the model were true? To find out you:


*

*Run $R-$ binomial trials of size $N_i$ with probability of success $p_i$ with $i=1,2,..,R$. These are your classes and defaults;

*Divide the number of successes by the number of trials for each class separately. This is $N_i p_i$ in the statistic. 

*Compute the statistic by comparing this simulation to the model and store it;

*Repeat steps 1 - 3 a large number of times e.g. 1,000,000;

*See what proportion of these simulated statistics are larger than the statistic you've computed by comparing your model to reality. If this proportion is smaller than say 5% or some other threshold value, then you have a statistically significant result.
On a side note: why are you using this statistic? Why not quantify how large the losses are from acting on this model as if it were true or how much money is lost through defaults and do steps 1-5 for that. This would seem more meaningful to me as this is what you're trying to avoid, correct?
