Cox proportional hazard Model $R^2 = 1$, good calibration - but Somers' $D = 0.16$

Question

I have analyzed the survival time of a currency Unit "not being recovered" (for a debt) by fitting a Cox P. regression model. Apparent $R^2=1$ is high, $D_{xy}=0.16$ . After bootstrap validating - validate(fit,b=50) (I have lots of observations so I only ran B=50) - Still getting high $R^2 = 0.997$ and $D_{xy}$ also the same as above. Does it make sense to have high $R^2$ and calibration looks ok, and not to narrow spread of predicted survival proportions. yet that low Somers' D?

Update: Context Consider a debt of $1000 that was not paid and the collection agency start collecting that debt and a followup time of 100 months . Recoveries/payments on that debt are done at different times say x is paid month=1 ,y is paid month=50 , z is paid month=60.

At each payment I add an observation with a column: event = 1(not censored) also adding a frequency weight of the recovered amount - say if that person paid $200 then frequency weight = 200, I also record the time say month=40. if the end of the followup period ends and outstanding debt is > 0 say 800 then we record that event = 0(censored) and set the frequency weight= 800, month = 80(since this debt appeared 20 month from the start of the observation time point)

Using the rms package i fitted a cox proportional hazard model:

fit <- cph( Surv(month,event) ~ x1 + x2 + x3 ,x=T, y=T,surv=T,time.inc = 1,weights=frequency , data = data )

due to the fact that the correlation may is strong I also tried.

fit2 <- robcov( fit, cluster = ID)

Which gave a bit a bit higher standard errors for the predictors but not that anything became not significant.

finally we get the calibration curve with:

calibrate(fit,b=40,u=90)

Can this explain the results that I am getting?

Answer 1

I suspect that your problem comes from a combination of using the amount of the payment as a frequency weight and having a large number of observations.

First, as Frank Harrell says in a comment on the question, the rationale for that weighting isn't at all clear. I think that is leading to a violation of the assumption of independent observations in the modeling. You might be thinking about modeling the recovery of each dollar amount this way, but the simultaneous collection of 800 dollars does not represent 800 independent recoveries of 1 dollar each, as that weighting implies. Furthermore, if there can be multiple payments from the same debtor, your model doesn't seem to take that additional lack of independence into account. Not accounting for a lack of independence among observations is a way to get strong apparent associations that aren't valid. I suspect that failure to take dependence among observations into account is also the reason for your apparent high calibration.

Second, the number of observations $n$ contributes to the calculation of the reported $R^2$ value; from the code for cph:

    logtest <- -2 * (f$loglik[1] - f$loglik[2])
    R2.max <- 1 - exp(2 * f$loglik[1]/n)
    R2 <- (1 - exp(-logtest/n))/R2.max
    

where f is the fitted model and the loglik values are the log-likelihoods at the initial [1] and final [2] parameter values. If your log-likelihoods didn't take lack of independence into account and you have a large number of observations, I suppose you could end up with a high apparent, but erroneous, $R^2$ (although I haven't though that through carefully). Look at those log-likelihood values.

The $D_{xy}$ calculation, unlike the likelihood-based calculations, does not take weighting into account, except insofar as weighting led to the final coefficient estimates. It just takes linear-predictor values based on the final model and the actual observation times and censoring indicators as arguments. My guess is that, in your case, the $D_{xy}$ of 0.16 (corresponding to a concordance of 0.58) better represents the discrimination quality of your model should you apply it to new cases.

Try modeling without that dollar weighting, perhaps including the amount of outstanding debt as a predictor, and allowing for multiple payments from the same debtor if they occur.