I would like to compare the curvature of two response surfaces, each of the form:
binomial ~ continuous variables1-5
I think it would be appropriate to use the effective degrees of freedom of a thin-plate spline fit to the data with generalized cross-validation as an estimate of the curvature. Is this correct? One of the surfaces is complex (effective df = 40), so parametric approaches are not practical here.
To test for a significant difference in curvature, I calculated the observed difference in effective df for thin-plate splines fit to the data using generalized cross-validation (using the Tps function in the Fields package in R). I then calculated this difference for 1000 samples with the binomial response variables randomly reassigned for each dataset to generate an empirical null distribution and then counted results >= the observed difference to get a one-tailed P-value.
My problem is this: the null distribution of effective degrees of freedom is incredibly large, spanning -10 to 120 df for the randomized datasets, even though the observed difference in df was only 20 df (40 and 20 df, respectively)!
Is this something I should be worried about? Why is GCV fitting such complex curves to randomized response surfaces? This problem seems to be inherent to resampling smoothing splines, GAMs, thin-plate splines, REML methods, and all variations of bootstrapping my various datasets: random binomial response surfaces often result in dramatically over-fitted splines.
Is there a better nonparametric way to compare curvature of two surfaces?