Is the distribution of the test statistic for the Hosmer-Lemeshow test $\chi^2$ in ''out of sample validation''? The Hosmer-Lemeshow test has some inconveniences and I am well aware of it. But assuming that I want to apply it then I read that (e.g. Applied logistic regression, by Hosmer and Lemeshow) the test statistic has $g-2$ degrees if freedom ($g$ is the number of partitions used to define the test statistic). 
Some of my colleagues however argue that this number of degrees of freedom may be dependent on whether you compute the test statistic on the training sample or on a validation sample. According to them the degrees of freedom to be used on a validation set would be $g$ in stead of $g-2$ because on the validation set you do not estimate any parameters.

Can anyone explain how I should determine the degrees of freedom for
  (1) the training set and (2) The validation set.

EDIT 8/10/2016

Because I found the existing answers ''confusing'' I added my own
  answer, I used simulations to make my point clear.

EDIT,
referring to the answer of @jwimberley and the comments below it:
in their 1980 paper, Hosmer and Lemeshow have proven that the test statistic (for a partition with $g$ groups) is $\chi^2(g-p-1) + \sum_{i=1}^p \lambda_i \chi^2(1)$.  
One can read that paper and you will find out that the term $\sum_{i=1}^p \lambda_i \chi^2(1)$ is a consequence of the fact that the partition is defined on predicted probabilities and therefore ''random''.  The first term can be explained by a similar reasoning as Pearson's GOF test. 
In a second step (see 1980 paper) Hosmer and Lemeshow show by simulations that the term $\sum_{i=1}^p \lambda_i \chi^2(1)$ can be approximated by a $\chi^2(p-1)$, and combining this we find that $\chi^2(g-p-1) + \sum_{i=1}^p \lambda_i \chi^2(1)$ is a $\chi^2$ with $g-p-1+p-1=g-2$ degrees of freedom. 
All these things will be confirmed by people that read that paper. 

The $g-2$ df of the HL statistic are widely known, so any simulation
  should be able to reproduce that.  If not then there is either a
  problem with the simulation or with the HL paper.  I have analysed the
  HL paper and I think there things are based on mathematical theorems and sound simulations.

I would like to find out how an out-of-sample test (or a test on a validation set) would change the results of Hosmer and Lemeshow, i.e. where in their proof/simulation would there be a difference using a validation set ?
EDIT 30/9/2016
@jwimberley 
If you do the Hosmer-Lemeshow test on a validation sample, then would I could expect is that in $\chi^2(g-p-1) + \sum_{i=1}^p \lambda_i \chi^2(1)$ the $p$ in the first term $\chi^2(g-p-1)$, where the $p$ is the consequence of the estimation of $p$ parameters, well for a validation sample there may be an difference on that term because in that validation sample you do not estimate $p$ parameters.  
However, for the second term $\sum_{i=1}^p \lambda_i \chi^2(1)$, a term that is there because of the use of predicted (random) probabilities to partition your validation set, well that term should also be there for the validation set.  This is what I meant when I said that the degrees of freedom where unusual in this thread: Dividing a sample based on the value of y would be problematic?.  
In your first comment and other comments below the answer in this linked thread, you denied that as you can read there. 
EDIT 7/10/2016, @jwimberley

 A: In their 1980 paper Hosmer D.W., S. Lemeshow, ''Goodness of fit tests for the multiple logistic regression model'', Communications in Statistics - Theory and Methods, Volume 9, 1980 -  Issue 10, the authors have proven that the test statistic (for a partition with $g$ groups) is $\chi^2(g-p-1) + \sum_{i=1}^p \lambda_i \chi^2(1)$.
In a second step they showed, using simulations that the term $\sum_{i=1}^p \lambda_i \chi^2(1)$ is a approximately $\chi^2(p-1)$ and as the sum of (independent) $\chi^2$ is also $\chi^2$ with degrees of freedom equal to the sum of the individual degrees of freedom 

they found that (1) their test statistic is exactly $\chi^2(g-p-1) +
 \sum_{i=1}^p \lambda_i \chi^2(1)$ and (2) their test statistic is
  approximately $\chi^2(g-2)$

Simulations under the Hosmer-Lemeshow conditions
As they have shown this formally, any simulation that is executed under their necessary conditions should find (approximately) a $\chi^2(8)$ when $g=10$. This is the case for the simulations below (in order to keep the answer ''readable'',  I inserted the ''helper'' functions at the botttom of this answer, these have to be executed first). 
The code contains comments to point to the major steps in the simulation. It is mainly a loop that is executed $N$ times, in each loop one has:


*

*Draw a training sample

*Estimate the logistic model

*Predict the probabilities using the estimated coefficients

*Compute the HL X2


The helper functions are defined at the end of my answer. 
simulateHosmerLemeshowX2<-function(N=5000, sampleSize, b0=-4) {

  x2HL<-vector(mode="numeric", length=N)

  for (i in 1:N) {

    # draw a training sample
    trainSample<-generateSample(sampleSize, b0)

    # train a logistic model
    logisticModel <- glm(y~x1+x2,family="binomial",data=trainSample$sample)

    #predict the probabilities using the estimates
    predictedPs<-predict(logisticModel, newdata = trainSample$sample, type = "response")

    # compute the Hosmer-Lemeshow statistic
    x2HL[i] <- hoslem.test(trainSample$sample$y,predictedPs)$statistic
  }


  p<-plotSimulatedX2(x2HL, N=N,title = "H0: model predicts probabilities well (Hosmer-Lemeshow)", df.chi=8)
  return(list(graph=p, N=N))    
}

simulateHosmerLemeshowX2(sampleSize=500, b0=-4)

One can execute this simulation, you will find a graph like the one below: 
The bars represent the simulated test statistic of Hosmer-Lemeshow, the smooth curve is a simulated $\chi^2(8)$. It seems to conform the findings of HL.  Note that the mean of the test statistic is close to 8 and that the mean of a chi-square is the number of degrees of freedom. 

Simulations for ''out-of-sample'' validation
In this section I simulate the out-of-sample situation.  The code and one result are below. 
The code contains comments to point to the major steps in the simulation. It is mainly a loop that is executed $N$ times, in each loop one has:


*

*Draw a sample

*Split it into a training and a validation sample

*Estimate the logistic model on the training sample

*Predict the probabilities using the estimated coefficients on the validation sample

*Compute the HL X2 on the validation sample


The helper functions are defined at the end of my answer. 
The graphs shows that the test statistic, compute on the validation sample, is not $\chi^2(8)$, the mean seems to indicate that one might conclde that it is $\chi^2$ with 13 df, but, as indicated in the graph, then the variance is not compatible with a chi-square ?
simulateHosmerLemeshowOutOfSampleX2B<-function(N=5000, sampleSize, b0) {

  x2HL<-vector(mode="numeric", length=N)


  for (i in 1:N) {

    # generate a sample
    fullSample<-generateSample(2*sampleSize, b0)

    # split the sample in two equal parts; one for training and one for validation
    idx<-sample(x = 1:(2*sampleSize), size = sampleSize)
    trainSample<-fullSample
    trainSample$sample <-fullSample$sample[idx,]
    trainSample$truePs <-fullSample$truePs[idx]

    validationSample<-fullSample
    validationSample$sample <-fullSample$sample[-idx,]
    validationSample$truePs <-fullSample$truePs[-idx]

    # train a logistic model on the training sample
    logisticModel <- glm(y~x1+x2,family="binomial",data=trainSample$sample)

    # use the trained model on the validation sample
    predictedPs<-predict(logisticModel, newdata = validationSample$sample, type = "response")

    # compute HL on the validation sample
    x2HL[i] <- hoslem.test(validationSample$sample$y,predictedPs)$statistic
  }


  p<-plotSimulatedX2(x2HL, N=N,title = "H0: model predicts probabilities well, Out of Sample \n Splitted sample", 
                     df.chi=12)

  return(list(graph=p, N=N)) 
}

simulateHosmerLemeshowOutOfSampleX2B(sampleSize=500, b0=-4, N=5000)


Annex: code for the helper functions.
library(ResourceSelection)
library(ggplot2)


generateSample<-function(sampleSize=100, b0=-4, b1=0.5, b2=3) {

  x1<-rnorm(sampleSize,mean=3)
  x2<-rnorm(sampleSize, mean=1)

  p <- 1/(1+exp(-(b0+b1*x1+b2*x2)))
  y <- rbinom(sampleSize,1,p)

  return(list(sample=data.frame(y=y,x1=x1,x2=x2), truePs=p))
}

plotSimulatedX2<-function(x2HL, N, title="", df.chi=8) {

  df1<-data.frame(x=x2HL, type="X2")
  df2<-data.frame(x=rchisq(n = 3*N, df = df.chi), type=paste("Chi^2(", df.chi,")") )
  df<-rbind(df1, df2)

  meanX2<-round(mean(df1$x),digits=2)
  varX2<-round(var(df1$x),digits=2)

  subTit<-paste("mean(X2)=",meanX2, " \nvar(X2)=", varX2,sep="")

  p<-ggplot() +
    geom_histogram(data = df1, aes(x=x, fill=type, y=..density..),binwidth=2, alpha=.5, position="identity") +
    geom_density(data = df2, aes(x=x, fill=type), alpha=.3)+
    xlim(c(0,50))+
    annotate("text",x=40,y=0.05,label=subTit, colour="red")+
    annotate("rect", xmin = 30, xmax = 50, ymin = 0.042, ymax = 0.058, alpha = .2)+
    ggtitle(label = title)+
    theme(plot.title = element_text(size = rel(1.5)))

  return(p)
}

A: In the original paper, Hosmer and Lemeshow used 8 df (with 10 decile groups) for estimating the model on development data. For validation data, you would use 9 df (df= # of groups - 1), based on what I've seen in literature.
Edit: I haven't got a reference to support the choice of 9 df.
