Stepwise logistic regression and sampling I am fitting a stepwise logistic regression on a set of data in SPSS.  In the procedure, I am fitting my model to a random subset that is approx. 60% of the total sample, which is about 330 cases.
What I find interesting is that every time I re-sample my data, I am getting different variables popping in and out in the final model.  A few predictors are always present in the final model, but others pop in and out depending on the sample.
My question is this.  What is the best way to handle this?  I was hoping to see the convergence of predictor variables, but that isn't the case.  Some models make much more intuitive sense from an operational view (and would be easier to explain to the decision makers), and others fit the data slightly better.  
In short, since variables are shuffling around, how would you recommend dealing with my situation?
Many thanks in advance.
 A: You might glance at the paper Stability Selection by Meinshausen and Buhlmann in J R Statist. Soc B (2010) 72 Part 4, and the discussion after it. They consider what happens when you repeatedly divide your set of data points at random into two halves and look for features in each half. By assuming that what you see in a one half is independent of what you see in the matching other half you can prove bounds on the expected number of falsely selected variables.
A: Don't use stepwise! See my paper
A: If you're going to use a stepwise procedure, don't resample.  Create one random subsample once and for all.  Perform your analysis on it.  Validate the results against the held-out data.  It's likely most of the "significant" variables will turn out not to be significant.
(Edit 12/2015: You can indeed go beyond such a simple approach by resampling, repeating the stepwise procedure, and re-validating: this will lead you into a form of cross-validation.  But in such a case more sophisticated methods of variable selection, such as ridge regression, the Lasso, and the Elastic Net are likely preferable to stepwise regression.)
Focus on the variables that make sense, not those that fit the data a little better.  If you have more than a handful of variables for 330 records, you're at great risk of overfitting in the first place.  Consider using fairly severe entering and leaving criteria for the stepwise regression.  Base it on AIC or $C_p$ instead of thresholds for $F$ tests or $t$ tests.
(I presume you have already carried out the analysis and exploration to identify appropriate re-expressions of the independent variables, that you have identified likely interactions, and that you have established that there really is an approximately linear relationship between the logit of the dependent variable and the regressors.  If not, do this essential preliminary work and only then return to the stepwise regression.)
Be cautious about following generic advice like I just gave, by the way :-).  Your approach should depend on the purpose of the analysis (prediction?  extrapolation?  scientific understanding?  decision making?) as well as the nature of the data, the number of variables, etc.
A: An important question is "why do why do you want a model with as few variables a possible?".  If you want to have as few variables as possible to minimize the cost of data collection for the operational use of your model, then the answers given by whuber and mbq are an excellent start.
If predictive performance is what is really important, then you are probably better off not doing any feature selection at all and use regularized logistic regression instead (c.f. ridge regression).  In fact if predictive performance was what was of primary importance, I would use bagged regularized logistic regression as a sort of "belt-and-braces" strategy for avoiding over-fitting a small dataset.  Millar in his book on subset selection in regression gives pretty much that advice in the appendix, and I have found it to be excellent advice for problems with lots of features and not very many observations.  
If understanding the data is important, then there is no need for the model used to understand the data to be the same one used to make predictions.  In that case, I would resample the data many times and look at the patterns of selected variables across samples to find which variables were informative (as mbq suggests, if feature selection is unstable, a single sample won't give the full picture), but I would still used the bagged regularized logistic regression model ensemble for predictions.
A: In general, there are two problems of feature selection:


*

*minimal optimal, where you seek for smallest set of variables that give you the smallest error

*all relevant, where you seek for all variables relevant in a problem


The convergence of predictor selection is in a domain of the all relevant problem, which is hell hard and thus requires much more powerful tools than logistic regression, heavy computations and a very careful treatment. 
But it seems you are doing the first problem, so you shouldn't worry about this. I can generally second whuber's answer, but I disagree with the claim that you should drop resampling -- here it won't be a method to stabilize feature selection, but nevertheless it will be a simulation for estimating performance of a coupled feature selection + training, so will give you an insight in confidence of your accuracy.
