1
$\begingroup$

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've encoded the unique values in source, destination and car model using one hot encoding, unique values in source and destination - 12 each
unique values in car model feature - 6.
sample size of data set - 300,000.
I've scaled the numeric data using Standard scaler.

I've tried Linear regression, Ridge, LASSO, Elastic net and Adaboot algorithm.

Results in RMSE:
Linear reg - 2.36
Ridge - 2.36
LASSO - 5.7
Elastic net - 6.3
Adaboost - 5.6

my doubt is why, Ridge, LASSO, Elastic Net and Adaboost doesn't perform well than Linear regression, and what might be the reason?

I just used different algorithms, just to understand how they perform in this data set (for academic purposes). I know, its difficult to comment, without examining the data.

Any comments or suggestions is appreciated.

$\endgroup$
  • $\begingroup$ did you tune the hyperparameters? $\endgroup$ – carlo Nov 23 '19 at 14:32
  • $\begingroup$ also, what base model did you use for adaboost? $\endgroup$ – carlo Nov 23 '19 at 14:32
  • $\begingroup$ How did you calculate the RMSE? On the training set, test set, cross-validatede etc.? $\endgroup$ – COOLSerdash Nov 23 '19 at 14:33
  • $\begingroup$ I've tuned the hyperparameters for Ridge, Lasso and Elastic net. And I've used Decision Tree Regressor(default) as base model for Adaboost @carlo $\endgroup$ – Gokul Elumalai Nov 23 '19 at 14:59
  • 1
    $\begingroup$ It just came to my mind that adaboost is technically a classification method. Are you sure you are using a boosting algorithm compatible with regression, like gradient boosting? Also, it's worth considering that boosting algorithms have to be tuned for good performances, even if they very easily outperform linear models without any tuning at all. $\endgroup$ – carlo Nov 26 '19 at 21:59
1
$\begingroup$

You cannot just throw regularization at a problem and assume it must improve the model.

Ridge is about on par with unregularized here. My suspicion then would be that the price is heavily non-linearly dependent on features, and when the other regularizers start aggressively chomping at the weights of some of the less powerful but valid predictors, you lose some information.

That said, this is guesswork based on very little data. It might not even be the fault of the model - it might, for example, be a bug in the feature engineering workflow. One thing to do would be to preprocess some dataset with the same features in the same way and see if the pattern sticks. Another would be to swap out the code doing the preprocessing for some library that does the same thing in its own way.

$\endgroup$
  • 1
    $\begingroup$ thanks @jkm, I can understand now, that there is a non-linear relationship that's why Lasso and Ridge failed, but why does Adaboost algorithm gives me a poor result, it should be able to capture non-linear relationship right. $\endgroup$ – Gokul Elumalai Nov 23 '19 at 15:34
  • $\begingroup$ Ridge didn't fail, it just didn't really change anything. And Adaboost is not inherently linear or not, it just glues a bunch of predictors together, but it's likely linear if all the others are too. If it is the model's fault, I suspect the first two lean towards overfitting, and the rest towards underfitting your dataset, but again - I'm flying blind here. $\endgroup$ – jkm Nov 23 '19 at 19:50
  • $\begingroup$ if linear reg. does better than boosting, then the problem is very much linear. $\endgroup$ – carlo Nov 26 '19 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.