0
$\begingroup$

For example, say I'm performing a regression to explain how much various factors (age, sex, diagnosis, procedure) affect total annual medical cost among patients.

There are thousands of diagnosis and procedure codes so it seems prudent to group them into similar categories based on average medical cost to avoid overfitting. There are various ways to do this, one would be using a decision tree.

Question: Am I "using up" degrees of freedom in my data if I apply the grouper algorithm followed by the regression on the same dataset? Should I apply the grouper on a random subset (or prior year's data) and then code the grouper categories into the holdout dataset?

$\endgroup$
6
  • 1
    $\begingroup$ Although it's unclear exactly how one would define "degrees of freedom" in this context, unquestionably any p-values produced by the regression would be wildly optimistic. $\endgroup$
    – whuber
    Oct 21, 2020 at 14:42
  • $\begingroup$ @whuber Yes - in a sense you're running two models with two sets of variables on the same data. To avoid overfitting you're actually making it worse? $\endgroup$
    – RobertF
    Oct 21, 2020 at 14:47
  • 1
    $\begingroup$ It's not necessarily worse, but caution is needed--as you obviously suspect, or you wouldn't be asking this question. I studied this issue in a simple situation once, with a single continuous regressor and a binary response. The decision tree made a single split, creating a $2\times 2$ table. The p-value for a test of independence in that table ranged by almost five orders of magnitude depending on where the split was made. Your problem is that employing the response (cost) in the clustering uses up a huge number of DF. If you cluster independently of the response, you're on firmer footing. $\endgroup$
    – whuber
    Oct 21, 2020 at 14:56
  • $\begingroup$ @whuber Good, that's what I thought. Surely someone's asked this question before, it seems like a basic Stats 101 issue. But I couldn't find anything here in Cross Validated. $\endgroup$
    – RobertF
    Oct 21, 2020 at 15:02
  • 1
    $\begingroup$ We do have some related questions, but I'm having trouble finding them. Your question reminds me a bit of stats.stackexchange.com/questions/38063 in the sense that the preliminary clustering based on the response variable seems akin to picking out the intervals in a process that maximally depart from randomness. $\endgroup$
    – whuber
    Oct 21, 2020 at 15:07

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.