I have a training data which after cleaning, wrangling has around 17k with more than 40 columns. The categorical columns are 35 and numerical 5. The categorical columns have value either 0 or 1. Now the thing is this training set is highly imbalanced as for every column I have, it has roughly 90% of the column as 0 and 10% like 1. I'm trying to predict prices using these columns. As I try to predict values using these columns I get a training score of 80% and validation of around 43-44 %. I have used Random Forest, XGBoost regressors for prediction. I have also tried cross-validation, hyperparameter tuning for these but the results don't seem to improve. Is it that the data is not right - being highly imbalanced and all or is it a problem with the model . How do I improve this crappy model ?

P.S : I can't share the data or code due to company policy

  • $\begingroup$ What metric are you useing? If you are getting that great a disparity between your training and test sets then it is likely an issue of over-fitting. Without seeing your exact code it is hard to say for certain, but some harsher cut-offs for hyper-parameter might be in order to avoid overfitting. How are you concocting your data partitions currently? $\endgroup$ – André.B Oct 28 '19 at 22:21
  • $\begingroup$ I'm using RMSE and R2 as my metric. I'm using K-fold cross validation where k=3/5/10 as my partitions. This was to prevent overfitting but sadly didn't work $\endgroup$ – harshit Oct 28 '19 at 22:54
  • $\begingroup$ Just to clarify, your test set is then randomly selected from the data? Have you tried increasing the range of the hyper-parameters? You could also try support vector regression. Lastly, there is always the potential that your covariates (a.k.a. features, inputs, etc.) are not up to the task (i.e. they aren't capable of predicting price accurately). $\endgroup$ – André.B Oct 29 '19 at 0:17
  • $\begingroup$ Yes , it is randomly selected from the data . I printed out the indices each time. $\endgroup$ – harshit Oct 29 '19 at 0:41
  • $\begingroup$ Covariates are not up to the task ....how do I figure that out? $\endgroup$ – harshit Oct 29 '19 at 0:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.