# Logistic Regression: Does my model selection process make sense?

This is kind of a broad question and so I am okay with broad or general answers. In fact, each of these could be their own individual questions, but I think it makes sense to ask them all. Even if you have answers to just one or two, I am happy with that. Basically I have made a model and approach and even have some results, but I just want to make sure that it's correct and that there are no gaps in the process. So, here goes:

Previous Criminal Activity as Predictor of Future Criminal Activity

(Note that this is an academic project and is not going to impact any real persons)

For simplicity, say that in my training set I have 100 individuals and 10 of them are convicted criminals. So my output variable (Y) has 90 zeroes (not convicted) and 10 ones (convicted). In my test set I have 10 individuals and one convicted criminal among them.

1. I have features of their behaviors and demographics and I want to figure out which features makes someone more likely to commit a crime. But I also want to break them into ranks, or buckets, so that the investigators can know who to target. For example, among the 90 that are not convicted, I only have enough money to pay investigators to research twenty of them. So how do I use the output to tell them which twenty are the riskiest?

2. So I put my training set into logistic regression with various features (some continuous and some categorical). For example, say the state that they live in, so I would have 49 categorical variables for the states. If I calculate VIFs for these and some states have high VIFs and others don't, does it mean that there is multicollinearity among them even though they are categorical? And does it make sense to pick and choose which categorical variables are to be removed? For example, does it make sense to proceed with 39 of the 50 states since I found that 11 had multicollinearity?

3. After that I do stepwise model selection. Lets say I get five out of thirty features with p-values being significant. So is it correct to assume that those features make most likely to perform criminal activity? And the coefficients describe how much impact (after transforming them back to linear estimates, of course)? Similarly to question (2.), does it make sense to drop parts of a categorical feature during this process? Or if you drop one, then you have to drop them all?

4. Once I do this, how do I use my output to "predict" which of the 90 are riskiest? My output would be 1's and 0's in order to make ROC curve, right? How do I convert that to some sort of predicted probability in between 0 and 1? So basically I would like to make five buckets, like 0-20%, 21-40%, 41-60%, 61-80%, and 81-100% based on predicted probabilities and I will tell the investigators to focus on the 81-100% first, and then if they have time, go to 61-80%, and so on.

5. In logistic regression, what is the difference between the training, validation, and test set? I am used to using a training and test set, but not sure about a validation set. Is that used to calculate the ROC curve?

6. Say that in my data I have too few 1's and a lot of 0's and I am getting really poor ROC curves (close to 0.5). Is there a sampling approach or other type of fix that I can perform to remedy this?

I hope that's not too broad and any guidance would be helpful. Thank you and please feel free to ask me for any clarifications!

1. The beauty of logistic regression is that it outputs probabilities. So just sort the subjects by their predicted probability of offending and pick the 20 greatest.

2. I don't know what you're asking. I think you have the vocabulary mixed up here. Multicollinearity isn't something you do; it's a condition of a dataset. A categorical feature comprises levels, not variables. What exactly do you mean by "VIF"?

3. To put it bluntly, stepwise model selection is an obsolete method. There are much better ways to do variable selection, such as the lasso, and you should be careful not to assume you need to do variable selection in the first place, because variable selection always carries the risk of throwing away useful information, and there are modeling techniques that can handle a lot of uninformative features, such as random forests (and lasso-regularized logistic regression, for that matter).

4. As I mentioned earlier, logistic regression produces probabilities, not 0s and 1s. Making an ROC curve just means coercing the probabilities to 0 and 1 with a varying threshold. But it seems more to the point to just give the investigators the complete list of subjects sorted by their probability of offending.

5. No, the idea of a validation set is just to allow you to tune some aspect of the models with the test set (such as, in the case of the lasso, the penalty size). Then you can examine the model's predictions in the validation set without any optimistic bias from overfitting the tuning procedure to the test set.

6. That depends entirely on how you're getting your data. Just don't throw away data you already have to balance your classes. That would be counterproductive. Edit: Oh, when you wrote "sampling" here, you probably meant "resampling". No, don't do that.

• Hey thank you so much. That was very helpful! Here are a few clarifying points: (2.) If I calculate VIFs for my variables (say ten categorical variables for one feature) and find high VIFs, that indicates multicollinearity, right? So I should take those out or make interaction variables? But I don't know how this applies to categorical variables. (6.) Let's say that my productivity (number of 1's) is so low that my results are bad. So basically I'm SOL, right? No way to "enhance" the data? – vdiddy Jun 30 '16 at 19:10
• Thanks again for the note. Made some edits to my original post as well. – vdiddy Jun 30 '16 at 19:16
• (2) I'm not familiar with VIF. But, it's probably better to use a method that can handle collinearity (e.g., ridge regression) than to try to remove it. (6) Don't use area under the ROC curve. It's not good for anything. Proper scoring rules are a good replacement for model comparison. But yeah, if you don't have enough 1s for your purposes, there's not much you can do short of collecting more data. – Kodiologist Jun 30 '16 at 19:22
• Okay thanks a lot. A very short and high-level explanation is that VIF values over 10 are indicators of multicollinearity. For a further deeper question on the categorical variables, I decided to ask another question. Maybe you can help with further clarity on that (minus the VIF parts) here: stats.stackexchange.com/questions/221540/… – vdiddy Jun 30 '16 at 19:25
• VIF is variance inflation factor, and indicates how much the stnd error of variable coefficients is "inflated" due to multicollinearity between variables. Sometimes researchers will drop one or more variables with high VIF values. VIF is not as useful in predictive models, where multicollinear variables are often retained in the model (usually with smaller coefficients) to increase predictive accuracy. – RobertF Jun 30 '16 at 19:32
1. Rank the 90 individuals by logit score from highest to lowest to obtain highest predicted probability of conviction. Examining the coefficients of the logistic regression will inform you which behaviors and demographics are most associated with conviction.

2. No - if you're building a purely predictive model (vs. an explanatory model), then avoid self-selecting predictor variables based on VIF scores. Instead, use one of several regularization techniques such as the lasso, ridge regression, or elastic net to select variables.

3. Avoid stepwise selection, since it often produces unreliable models based on biased tests at each step. See (2).

4. See (1) - individuals can be ranked by their logit scores.

5. Training, validation, and test (or holdout) datasets are frequently used in predictive modeling to compare different modeling methodologies (for example, logistic regression vs. decision trees). Models are built or "trained" using training data and then prediction errors for the models are computed on the validation data. Holdout data is sometimes used to assess generalization error after final models, since even with training/validation some degree of over fitting can occur when multiple models are calibrated and compared. For small datasets, I think 5 or 10-fold cross validation will serve you better. The ROC is a means of assessing logistic regression and binary classifiers by plotting true positive rate against false positive rate. Another is the area under the ROC curve, or c-statistic. In your case the ROC & c-statistics from competing models could be plotted with the validation data.

6. Yes - one approach for modeling sparse binary data is to use one of the aforementioned regularized logistic regression techniques.

• Hey thanks a lot. Appreciate your help. I know about LASSO, Ridge, etc. but still a little confused about how they can enhance my response variables? Also, when you say predictive versus explanatory, are you referring to forecasting versus modeling the data as it is? – vdiddy Jun 30 '16 at 19:34
• Yes, in part. Predictive modeling seeks to minimize prediction error & is frequently used for forecasting. Explanatory modeling is more concerned with explaining cause-effect paths between explanatory variables and the response rather than making future predictions. The $R^2$ statistic with linear regression is an explanatory tool for measuring goodness of fit. Predictive models minimize bias + variance combined, while with explanatory models researchers want to minimize bias in regression coefficients, even at the cost of higher variance and less predictive accuracy. – RobertF Jun 30 '16 at 19:54

You number of criminals relative to the number of non-criminals will be very low. This can cause some issues in certain cases, You may be better off using Poisson regression to estimate counts and relative risks rather than log odds (the output from logistic regression that can be converted to probabilities). Also, on the multicolinearity issue. This could result in an underestimation of significance, but if your model is theory-driven and replicating something that is "tried and true" - no big deal. If you are trying to tests a new hypothesis, is there a sufficient reason to group certain states together that seem to have colinearity? Stratify by certain attributes and reduce the number of independent variables? That would make a simpler model which is generally better if it is still useful for your project. Also, when you convert back to probabilities, you will not be able to break them into quintiles like you imagine. Your highest predicted probability of committing a crime in your dataset may be 20%. you will have to split you probibilites after thay are calculated into groups. I would suggest just splitting into 2 groups, maybe three at most. With the stepwise, don't rely on that. the number of variables you are dealing with is small enough that you can evaluate what should be put in and what should not. The only reason to allow the computer to so this for you is if there are so many variables it becomes to inefficient to do otherwise. You can use this to HELP you decide the variables and evaluate quickly the impact of including diffferent combinations. But in the end it is your hypothesis and reaoning that has to stand the test, not what the computer picks. Stepwise regression can be great for breaking into your data and looking at possible variables to include by comparing the resulting models, but in the end, you need to present good reasons for including the variables you do.

• Welcome to our site! You might want to consider breaking this post up into smaller paragraphs. Also it's worth noting that, even if there were too many variables to deal with them manually, stepwise regression is widely regarded as obsolete and dangerous, so should be avoided even in these circumstances. – Silverfish Jul 1 '16 at 0:16
1. As pointed out by others, logistic regression tries to estimate the probability of the outcome occurring so its outputs can be these probability estimates. If whatever software you are using is just outputting {0,1} for each case it may be splitting based on some cut off probability, say if it estimates that for record 1 the probability of future offending is more than 50% then it outputs 1, and if less than 50% it outputs 0. There may be a way to ask for the probability estimates explicitly, or the log odds.

2. With 100 samples you're not going to be able to use 49 categorical variables. There's likely some states which don't have any records in so how could you possibly guess the impact of being from that state? The same is actually true for any state with very few records. If you had a state which had, for example, at least one sample with no future criminal activity and no samples with future criminal activity (I would be quite surprised if you didn't have this) then the model will predict that no-one from that state will ever commit a crime in the future. Obviously this isn't true and highlights just a particularly obvious failure to generalise from the sampled data to the population overall. It's (sort of*) an example of what we call overfitting, which you should definitely make sure you understand when trying to draw conclusions about the total population from your model.

3. There is no simple answer to the question 'how to I determine which variables are relevant and which are not?' There are particular problems with using p-values on tests of 'coefficient = 0' and I would agree with what others have said that a method called the lasso is better (it will actually set coefficients to exactly 0 if it deems them not sufficiently predictive) if your goal is prediction. I don't know how well I can explain this distinction in a short answer but basically, if you're going to use the model to identify people at higher risk of future offending (prediction) then that should be fine. If you want to know what policies should be implemented to reduce crime rates (or, what causal influence do the things you've measured have), the model can't help you there. There are totally different techniques under the heading 'Causal Inference' which may be appropriate depending on what your analysis is going to be used for.