Logistic Regression: Does my model selection process make sense?

Question

This is kind of a broad question and so I am okay with broad or general answers. In fact, each of these could be their own individual questions, but I think it makes sense to ask them all. Even if you have answers to just one or two, I am happy with that. Basically I have made a model and approach and even have some results, but I just want to make sure that it's correct and that there are no gaps in the process. So, here goes:

Previous Criminal Activity as Predictor of Future Criminal Activity

(Note that this is an academic project and is not going to impact any real persons)

For simplicity, say that in my training set I have 100 individuals and 10 of them are convicted criminals. So my output variable (Y) has 90 zeroes (not convicted) and 10 ones (convicted). In my test set I have 10 individuals and one convicted criminal among them.

1. I have features of their behaviors and demographics and I want to figure out which features makes someone more likely to commit a crime. But I also want to break them into ranks, or buckets, so that the investigators can know who to target. For example, among the 90 that are not convicted, I only have enough money to pay investigators to research twenty of them. So how do I use the output to tell them which twenty are the riskiest?

2. So I put my training set into logistic regression with various features (some continuous and some categorical). For example, say the state that they live in, so I would have 49 categorical variables for the states. If I calculate VIFs for these and some states have high VIFs and others don't, does it mean that there is multicollinearity among them even though they are categorical? And does it make sense to pick and choose which categorical variables are to be removed? For example, does it make sense to proceed with 39 of the 50 states since I found that 11 had multicollinearity?

3. After that I do stepwise model selection. Lets say I get five out of thirty features with p-values being significant. So is it correct to assume that those features make most likely to perform criminal activity? And the coefficients describe how much impact (after transforming them back to linear estimates, of course)? Similarly to question (2.), does it make sense to drop parts of a categorical feature during this process? Or if you drop one, then you have to drop them all?

4. Once I do this, how do I use my output to "predict" which of the 90 are riskiest? My output would be 1's and 0's in order to make ROC curve, right? How do I convert that to some sort of predicted probability in between 0 and 1? So basically I would like to make five buckets, like 0-20%, 21-40%, 41-60%, 61-80%, and 81-100% based on predicted probabilities and I will tell the investigators to focus on the 81-100% first, and then if they have time, go to 61-80%, and so on.

5. In logistic regression, what is the difference between the training, validation, and test set? I am used to using a training and test set, but not sure about a validation set. Is that used to calculate the ROC curve?

6. Say that in my data I have too few 1's and a lot of 0's and I am getting really poor ROC curves (close to 0.5). Is there a sampling approach or other type of fix that I can perform to remedy this?

I hope that's not too broad and any guidance would be helpful. Thank you and please feel free to ask me for any clarifications!

Answer 1

1. The beauty of logistic regression is that it outputs probabilities. So just sort the subjects by their predicted probability of offending and pick the 20 greatest.

2. I don't know what you're asking. I think you have the vocabulary mixed up here. Multicollinearity isn't something you do; it's a condition of a dataset. A categorical feature comprises levels, not variables. What exactly do you mean by "VIF"?

3. To put it bluntly, stepwise model selection is an obsolete method. There are much better ways to do variable selection, such as the lasso, and you should be careful not to assume you need to do variable selection in the first place, because variable selection always carries the risk of throwing away useful information, and there are modeling techniques that can handle a lot of uninformative features, such as random forests (and lasso-regularized logistic regression, for that matter).

4. As I mentioned earlier, logistic regression produces probabilities, not 0s and 1s. Making an ROC curve just means coercing the probabilities to 0 and 1 with a varying threshold. But it seems more to the point to just give the investigators the complete list of subjects sorted by their probability of offending.

5. No, the idea of a validation set is just to allow you to tune some aspect of the models with the test set (such as, in the case of the lasso, the penalty size). Then you can examine the model's predictions in the validation set without any optimistic bias from overfitting the tuning procedure to the test set.

6. That depends entirely on how you're getting your data. Just don't throw away data you already have to balance your classes. That would be counterproductive. Edit: Oh, when you wrote "sampling" here, you probably meant "resampling". No, don't do that.

Answer 2

1. Rank the 90 individuals by logit score from highest to lowest to obtain highest predicted probability of conviction. Examining the coefficients of the logistic regression will inform you which behaviors and demographics are most associated with conviction.

2. No - if you're building a purely predictive model (vs. an explanatory model), then avoid self-selecting predictor variables based on VIF scores. Instead, use one of several regularization techniques such as the lasso, ridge regression, or elastic net to select variables.

3. Avoid stepwise selection, since it often produces unreliable models based on biased tests at each step. See (2).

4. See (1) - individuals can be ranked by their logit scores.

5. Training, validation, and test (or holdout) datasets are frequently used in predictive modeling to compare different modeling methodologies (for example, logistic regression vs. decision trees). Models are built or "trained" using training data and then prediction errors for the models are computed on the validation data. Holdout data is sometimes used to assess generalization error after final models, since even with training/validation some degree of over fitting can occur when multiple models are calibrated and compared. For small datasets, I think 5 or 10-fold cross validation will serve you better. The ROC is a means of assessing logistic regression and binary classifiers by plotting true positive rate against false positive rate. Another is the area under the ROC curve, or c-statistic. In your case the ROC & c-statistics from competing models could be plotted with the validation data.

6. Yes - one approach for modeling sparse binary data is to use one of the aforementioned regularized logistic regression techniques.

Answer 3

You number of criminals relative to the number of non-criminals will be very low. This can cause some issues in certain cases, You may be better off using Poisson regression to estimate counts and relative risks rather than log odds (the output from logistic regression that can be converted to probabilities). Also, on the multicolinearity issue. This could result in an underestimation of significance, but if your model is theory-driven and replicating something that is "tried and true" - no big deal. If you are trying to tests a new hypothesis, is there a sufficient reason to group certain states together that seem to have colinearity? Stratify by certain attributes and reduce the number of independent variables? That would make a simpler model which is generally better if it is still useful for your project. Also, when you convert back to probabilities, you will not be able to break them into quintiles like you imagine. Your highest predicted probability of committing a crime in your dataset may be 20%. you will have to split you probibilites after thay are calculated into groups. I would suggest just splitting into 2 groups, maybe three at most. With the stepwise, don't rely on that. the number of variables you are dealing with is small enough that you can evaluate what should be put in and what should not. The only reason to allow the computer to so this for you is if there are so many variables it becomes to inefficient to do otherwise. You can use this to HELP you decide the variables and evaluate quickly the impact of including diffferent combinations. But in the end it is your hypothesis and reaoning that has to stand the test, not what the computer picks. Stepwise regression can be great for breaking into your data and looking at possible variables to include by comparing the resulting models, but in the end, you need to present good reasons for including the variables you do.

Answer 4

1. As pointed out by others, logistic regression tries to estimate the probability of the outcome occurring so its outputs can be these probability estimates. If whatever software you are using is just outputting {0,1} for each case it may be splitting based on some cut off probability, say if it estimates that for record 1 the probability of future offending is more than 50% then it outputs 1, and if less than 50% it outputs 0. There may be a way to ask for the probability estimates explicitly, or the log odds.

2. With 100 samples you're not going to be able to use 49 categorical variables. There's likely some states which don't have any records in so how could you possibly guess the impact of being from that state? The same is actually true for any state with very few records. If you had a state which had, for example, at least one sample with no future criminal activity and no samples with future criminal activity (I would be quite surprised if you didn't have this) then the model will predict that no-one from that state will ever commit a crime in the future. Obviously this isn't true and highlights just a particularly obvious failure to generalise from the sampled data to the population overall. It's (sort of*) an example of what we call overfitting, which you should definitely make sure you understand when trying to draw conclusions about the total population from your model.

3. There is no simple answer to the question 'how to I determine which variables are relevant and which are not?' There are particular problems with using p-values on tests of 'coefficient = 0' and I would agree with what others have said that a method called the lasso is better (it will actually set coefficients to exactly 0 if it deems them not sufficiently predictive) if your goal is prediction. I don't know how well I can explain this distinction in a short answer but basically, if you're going to use the model to identify people at higher risk of future offending (prediction) then that should be fine. If you want to know what policies should be implemented to reduce crime rates (or, what causal influence do the things you've measured have), the model can't help you there. There are totally different techniques under the heading 'Causal Inference' which may be appropriate depending on what your analysis is going to be used for.

4. Has been answered elsewhere.

5. Splitting the data like this is (mostly) an attempt to avoid the problem of overfitting. If the predictions made on the test set (which the model didn't get to 'learn' from) are accurate, you can be more satisfied that they will continue to be accurate on the population at large. Again, your small sample size is going to make it very hard to be satisfied in this way (well, to even decide how satisfied you are). There are lots of more advanced options to help with this problem such as resampling techniques (cross-validation and/or bootstrapping), incorporating prior information via a bayesian model, or better yet, finding explicit generalisation error bounds from VC theory. Of course, probably none of these suggestions is helpful if it's going to take too long for you to understand how to use them and what they tell you. Take the thousands of journal pages filled with attempts to solve this problem satisfactorily as evidence that it's a very hard problem in general and in the face of that difficulty, the approach of virtually all statisticians is a great deal of humility and modesty in making claims about reality based on an analysis like this.

6. Class imbalances (more 0s than 1s) won't affect your ROC curve directly, if it's pretty close to the diagonal that just shows that the inputs can't give a particularly accurate prediction of the outcome. Class imbalance can make your model worse and there are techniques like SMOTE, noisy PCA etc. Try googling 'class imbalance problem'. It's certainly much less of a problem when you use a measure of model 'goodness' which is insensitive to class imbalances (such as area under ROC, F1 score, cohen's kappa) instead of raw accuracy. I would recommend against spending effort here.

*It could also be seen as failing to take into account sampling error, but I don't really think a discussion of inference vs induction is particularly useful here, and I'm trying to emphasise the dangers of pretending (or not realising that you are pretending) that 'statistics' has distilled pure (or at least 95%) certainty out of uncertainty.