I am trying to predict crimes (san francisco) using machine learning algorithms. It is a multi class classification problem with unbalanced data.
I took sample of data ranging from years 2010 to 2015 with 10 crimes (10 classes with varying distribution). I kept data from 2010 to 2014 for training and 2015 for testing.
Since it is unbalanced I did under sampling on majority class and over sampling on almost five minority classes in my training set. I used random forest as my primary algorithm.
I tried to predict test set with my model. My test set is still unbalanced but I get poor accuracy. I also tried adaboost and multinomial logistic regression, but to no use.
I did 10-fold stratified sampling on the training set. I got good accuracy but it is of no use, since I duplicated the minority classes as the process of over sampling.
I also tried log-loss, f1_score (weighted, micro and macro) as my performance metrics, but I didn't get a satisfying result.
Question: How can I proceed further? What else can I try?