This is kind of a broad question and so I am okay with broad or general answers. In fact, each of these could be their own individual questions, but I think it makes sense to ask them all. Even if you have answers to just one or two, I am happy with that. Basically I have made a model and approach and even have some results, but I just want to make sure that it's correct and that there are no gaps in the process. So, here goes:
Previous Criminal Activity as Predictor of Future Criminal Activity
(Note that this is an academic project and is not going to impact any real persons)
For simplicity, say that in my training set I have 100 individuals and 10 of them are convicted criminals. So my output variable (Y) has 90 zeroes (not convicted) and 10 ones (convicted). In my test set I have 10 individuals and one convicted criminal among them.
I have features of their behaviors and demographics and I want to figure out which features makes someone more likely to commit a crime. But I also want to break them into ranks, or buckets, so that the investigators can know who to target. For example, among the 90 that are not convicted, I only have enough money to pay investigators to research twenty of them. So how do I use the output to tell them which twenty are the riskiest?
So I put my training set into logistic regression with various features (some continuous and some categorical). For example, say the state that they live in, so I would have 49 categorical variables for the states. If I calculate VIFs for these and some states have high VIFs and others don't, does it mean that there is multicollinearity among them even though they are categorical? And does it make sense to pick and choose which categorical variables are to be removed? For example, does it make sense to proceed with 39 of the 50 states since I found that 11 had multicollinearity?
After that I do stepwise model selection. Lets say I get five out of thirty features with p-values being significant. So is it correct to assume that those features make most likely to perform criminal activity? And the coefficients describe how much impact (after transforming them back to linear estimates, of course)? Similarly to question (2.), does it make sense to drop parts of a categorical feature during this process? Or if you drop one, then you have to drop them all?
Once I do this, how do I use my output to "predict" which of the 90 are riskiest? My output would be 1's and 0's in order to make ROC curve, right? How do I convert that to some sort of predicted probability in between 0 and 1? So basically I would like to make five buckets, like 0-20%, 21-40%, 41-60%, 61-80%, and 81-100% based on predicted probabilities and I will tell the investigators to focus on the 81-100% first, and then if they have time, go to 61-80%, and so on.
In logistic regression, what is the difference between the training, validation, and test set? I am used to using a training and test set, but not sure about a validation set. Is that used to calculate the ROC curve?
Say that in my data I have too few 1's and a lot of 0's and I am getting really poor ROC curves (close to 0.5). Is there a sampling approach or other type of fix that I can perform to remedy this?
I hope that's not too broad and any guidance would be helpful. Thank you and please feel free to ask me for any clarifications!