Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite
2

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1.

My original modelling dataset has 100,000 observations - which I have divided into two samples - a training sample with 80,000 observations and a test sample of 20,000 observations. The samples were created randomly, maintaining the same proportion of 1 to 0 in both the samples (using the SURVEYSELECT procedure in SAS and $y$ as STRATA).

Let us assume that the percentage of observations with the value 1 for $y$ in both the samples is 10%.

I use the LOGISTIC procedure in SAS on the training sample to arrive at variables $x_1$ - $x_5$ which constitute my final model. The p-values associated with the Wald Chi Square are all <0.0001, which indicate that they are significant at the 99.99% confidence level.

However, when I run the LOGISTIC procedure on the test sample, using only $x_1$ - $x_5$ as independent variables, one of them say $x_4$ is no longer significant in the regression outputs - that is, the associated p-value is 0.6.

Does this mean my sampling is not proper? Or the model which I obtain from the training sample 'overfits' the data? Or both?

Should I be worried about this and ensure that the final variables I choose are significant in both the samples or this is not an issue in general?

share|improve this question
How did you arrive at the variables $x_1$ to $x_5$. I'm guessing it was through some stepwise or all subsets method. – Peter Flom Jan 17 at 11:34
@PeterFlom Yes, primarily STEPWISE but I also use FORWARD and BACKWARD selection to check which variables are chosen. I also look for multicollinearity using VIF (PROC REG) and generally ensure that the final variables have a value less than 2 (3 at max). – Mozan Sykol Jan 17 at 11:48
How many variables did you have to begin with? If it was a thousand or so, I would not be surprised that even the nominal 0.01% is not that significant. – StasK Jan 18 at 12:49
@StasK I had around 100 variables to start with. – Mozan Sykol Jan 18 at 18:36

2 Answers

up vote 6 down vote accepted

You haven't overfit your model, what you've done is demonstrate (again) that stepwise, forward and backward methods don't work well for this type of task. (Although it was good that you used a training and test set, this let you see that these methods can find things that aren't there).

Model selection is a big topic and has often been discussed, both here and elsewhere. I would generally advise against any automatic variable selection scheme, but if you must use one, I suggest LASSO or LAR. Since you are using SAS, you can find both of these in GLMSELECT. Although this is intended for models that can be fit with PROC GLM, I have have good results using it for logistic models and then testing the resulting models further in LOGISTIC.

share|improve this answer
Thanks, but let's say that I choose the variable using 'judgment' rather than any selection method but only looking at results in the training sample. Later I find that it does not work in the test sample. What is the (statistical) conclusion, other than the fact that my judgment might be bad? – Mozan Sykol Jan 17 at 13:46
Personally, I am not a fan of deleting important variables from the model just because they are not significant. You can learn from negative results; also, some variables are important as covariates. – Peter Flom Jan 17 at 19:30
1  
I am accepting this answer as the better of the two - but it still does not answer my question really. What I was looking for is not how to choose my model variables, but once chosen (for argument's sake, let's say given by God) and I am in a situation as I described - then how do I guess what has gone wrong, assuming something has gone wrong, and start correcting my mistakes? In my specific case, I just dropped the offending variables without loss of discriminatory or predictive power. – Mozan Sykol Jan 23 at 8:18
up vote 1 down vote

It can mean too many things, it may mean your data miss critical observations, it may mean that the problem you are facing is not as solvable as you thought (e.g. trying to fit a nonlinear determinstic model to some iid data stream), and yes, sure, for the case you demonstrated, it can also mean your model is over-, or more properlly speaking, ill-fitted.

Black-box modelling is always hard, ill-fitting or overfitting is a common outcome when one try to blindly fit some randomly-chosen, standard model structures to data when they have basically no idea of the phyiscs of the practical system. For many practical problems one has to has some decent grasp of the underlining physics of the system one try to model to develop a proper model, there is not a "default" method or standardized routine for modelling and no short-cuts, and thats why experienced modellers/data miners are all very highly rewarded in industry.

Edited: As for lasso method, personally I dont believe lasso method is that useful, if the problem can be solved by lasso, it certainly can be solved by many other regression methods, althrough at various degrees, the OP's problem (e.g. with 20000 randomly picked test samples) is not likely to be the case.

share|improve this answer
You make a broad statement that you don't believe lasso is that useful and that other methods can work just as well. You don't back this up with anything, you don't say which other methods; lasso has a lot of documentation of its effectiveness, and many other methods have lots of documentation of their ineffectiveness, at least for the tasks for which LASSO was designed. Have you evidence for your assertions? – Peter Flom Jan 18 at 12:08
@Peter Flom I dont think it is broad, if your regressor give you a p-value of 0.6 with 20000 test samples, then I seriously doubt the problem lies in the regression methods he used, lasso or not, much more likely scenarios are either OP miss critical observations in his data set or OP try to "invent" some determinstic knowledge/trend from some largely stochastic process/random data and failed as expected. – user55647 Jan 18 at 12:41
@user55647 actually, it's a reasonably "simple" case where I am trying to predict if a potential customer would respond to a marketing campaign. One of the standard metrics we use is Gini (same as Somer's D in this case) to measure the discriminatory power of the model - and surprisingly, the Gini increases if I remove the problematic variable from the model. – Mozan Sykol Jan 18 at 18:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.