1
$\begingroup$

The question is rather theoretical than practical. Suppose, we want to evaluate one's chances of non-paying back a loan. We assume, that these chances depend on some set of covariates. One of the covariates is first two digits of ZIP code, let's call it zip2.

We have a set of observations (loans) for some period of time, where for each loan we observe target variable y = 1, if loan is paid back, and y = 0 otherwise. Also, we have values of independent variable x = zip2. Let's say that # of observation is satisfactory, for example, 2000.

Now, we are building a logit model in order to be able to predict the chances for the incoming loans to be (or not to be) paid back. Suppose, we run some software package, like R or Stata to estimate the model with our dataset. In the result we see, that our independent variable zip2 is not significant in the model.

Then, we try to analyze our dataset more closely. We see that percentage of "bad" outcomes for zip2 = "11" and zip2 = "73" is virtually the same, say 15%. So, we decide to make one category out of these 2. We also examine the other values of the independent variables, and categorize them in a similar fashion, so, in total, we end up with 6-7 categories.

Now, when we run estimation for the model with "categorized" variable zip2, we see that it becomes significant!

So, my question is: Is this correct to do such categorization? I mean, we group independent variable by outcomes of dependent variable, and then we try to predict the outcome. It seems like a vicious circle, and I feel that it is not correct at all.

This example is taken from a real situation, existing in a risk department of one company. If this is wrong, I just want to point out that such approach to model building is not acceptable and they're simply making wrong decisions.

I already consulted a professor in statistics, and he said that this was wrong to group independent variables with respect to outcomes of dependent variable. But, I want to find some theoretical proofs, why such grouping is wrong. Is there any book or article, which explains inconsistency of such approach?

Or, maybe it is consistent?

$\endgroup$
3
  • $\begingroup$ A common critique is that hindsight is 20-20. It would be important to know how well this categorization system would perform in predicting results for brand-new data. Try looking up cross-validation and seeing the rationales people provide for doing it. $\endgroup$
    – rolando2
    Dec 30, 2013 at 13:03
  • $\begingroup$ Using the outcome variable in order to reduce the number of parameters in a model is generally called "data dredging". (See e.g. the famous book "Regression Modeling Strategies" by Frank Harrell). Btw, 2000 observations is not much if you use predictors with 100 levels. $\endgroup$
    – Michael M
    Dec 30, 2013 at 13:25
  • $\begingroup$ @Michael Mayer. As I told, it is theoretical question, not practical. # of observation does not matter here, I put 2000 just for example. I'll take a look on some articles about "data dredging", thank you for suggestion! $\endgroup$
    – Oleksandr
    Dec 30, 2013 at 15:10

1 Answer 1

4
$\begingroup$

Besides the severed data dredging problems caused by such a strategy, I question the use of zip code in this way. I suggest replacing zip code with the median family income (from US Census data) per zip code, and modeling the effect of income as a continuous variable, non-linearly using regression splines.

Back to the data dredging problem, Stephen Senn has developed a nice analogy with wanting to place a bet at the horse track on the winner.

$\endgroup$
3
  • $\begingroup$ Well, it's not a question about variables choice. I bet that median income would explain outcomes better in this case. I used zip2 only for illustration of the grouping strategy, and that's it. $\endgroup$
    – Oleksandr
    Dec 30, 2013 at 14:54
  • $\begingroup$ As I understand, data dredging is something bad. Am I correct? And, in this case, we may not do such categorization, right? $\endgroup$
    – Oleksandr
    Dec 30, 2013 at 15:27
  • $\begingroup$ Data dredging badly damages statistical inference (hypothesis testing and confidence intervals) and in most cases does not help with predictive accuracy. The process outlined in the OP only seems to save parameters; that is an illusion. There are effectively extra parameters used in the process of pooling estimates. $\endgroup$ Dec 31, 2013 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.