1
$\begingroup$

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've encoded the unique values in source, destination and car model using one hot encoding, unique values in source and destination - 12 each
unique values in car model feature - 6.
sample size of data set - 300,000.
I've scaled the numeric data using Standard scaler.

I've tried Linear regression, Ridge, LASSO, Elastic net and Adaboot algorithm.

Results in RMSE:
Linear reg - 2.36
Ridge - 2.36
LASSO - 5.7
Elastic net - 6.3
Adaboost - 5.6

my doubt is why, Ridge, LASSO, Elastic Net and Adaboost doesn't perform well than Linear regression, and what might be the reason?

I just used different algorithms, just to understand how they perform in this data set (for academic purposes). I know, its difficult to comment, without examining the data.

Any comments or suggestions is appreciated.

Improve this question
$\endgroup$
7
  • $\begingroup$ did you tune the hyperparameters? $\endgroup$
    – carlo
    Commented Nov 23, 2019 at 14:32
  • $\begingroup$ also, what base model did you use for adaboost? $\endgroup$
    – carlo
    Commented Nov 23, 2019 at 14:32
  • $\begingroup$ How did you calculate the RMSE? On the training set, test set, cross-validatede etc.? $\endgroup$
    – COOLSerdash
    Commented Nov 23, 2019 at 14:33
  • $\begingroup$ I've tuned the hyperparameters for Ridge, Lasso and Elastic net. And I've used Decision Tree Regressor(default) as base model for Adaboost @carlo $\endgroup$
    – chqdrian
    Commented Nov 23, 2019 at 14:59
  • 1
    $\begingroup$ It just came to my mind that adaboost is technically a classification method. Are you sure you are using a boosting algorithm compatible with regression, like gradient boosting? Also, it's worth considering that boosting algorithms have to be tuned for good performances, even if they very easily outperform linear models without any tuning at all. $\endgroup$
    – carlo
    Commented Nov 26, 2019 at 21:59

1 Answer 1

Reset to default
1
$\begingroup$

You cannot just throw regularization at a problem and assume it must improve the model.

Ridge is about on par with unregularized here. My suspicion then would be that the price is heavily non-linearly dependent on features, and when the other regularizers start aggressively chomping at the weights of some of the less powerful but valid predictors, you lose some information.

That said, this is guesswork based on very little data. It might not even be the fault of the model - it might, for example, be a bug in the feature engineering workflow. One thing to do would be to preprocess some dataset with the same features in the same way and see if the pattern sticks. Another would be to swap out the code doing the preprocessing for some library that does the same thing in its own way.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ thanks @jkm, I can understand now, that there is a non-linear relationship that's why Lasso and Ridge failed, but why does Adaboost algorithm gives me a poor result, it should be able to capture non-linear relationship right. $\endgroup$
    – chqdrian
    Commented Nov 23, 2019 at 15:34
  • $\begingroup$ Ridge didn't fail, it just didn't really change anything. And Adaboost is not inherently linear or not, it just glues a bunch of predictors together, but it's likely linear if all the others are too. If it is the model's fault, I suspect the first two lean towards overfitting, and the rest towards underfitting your dataset, but again - I'm flying blind here. $\endgroup$
    – jkm
    Commented Nov 23, 2019 at 19:50
  • $\begingroup$ if linear reg. does better than boosting, then the problem is very much linear. $\endgroup$
    – carlo
    Commented Nov 26, 2019 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.