Data set:

I'm working on a linear regression problem where my train set $X$ is of shape $(703 557, 53)$. Each row is a client's features, which could be its age, its gender, how many calls we received from that client during a month... The data set is transformed in the sense it's not raw data but some categorical variables have been "one-hot encoded" as the number of times a category appeared during a month. For example if the client visited a private center 2 times during january we'll have a row for that client for that january with the feature "center_visited_private" taking value 2, and if the client visite a public center 3 times in june then we'd have another row with the feature "center_visited_public" taking value 3, with other features obviously. My target is the price the client would pay.

So I have data over $5$ years ($2014 - 2018$) and my cross-validation strategy is to train on $2014-2016$ and select models based on performance on $2017$ data. I kept the $2018$ as test set.

Issues with Data set:

Since my models are doing poorly, I figured out that some issues with the data set may be the root of the problem. I could count these issues:

  • There is some redundancy since some features could keep the same value for a subset of clients. Which could lead the models to learn patterns over a subset of clients but not others. Or to just be overwhelmed with the different patterns related to the different clients and learn nothing.
  • The data isn't i.i.d as we have historical data of each client so the accumulation of past behaviour could lead to the present observed behaviour.
  • The result of the "one-hot encoding" led to a sparse matrix.

Models Results:

The RMSE and MAE baselines are respectively: 877 and 201.

  • Random Forests: n_estimators = 500, criterion='squared error', max_features='sqrt' => It took quite some time to train, around 45 min, and it returned a $RMSE=900$ and a $MAE=220$. Slightly negative $R^{2}$ (nearly 0 but negative).
  • Linear Regression, Ridge, Lasso (with different regularization parameters) => Trained pretty fast, not more than 10 sec, $RMSE=877$, $MAE=201$. Slightly negative $R^{2}$ (nearly 0 but negative).
  • Huber Regressor, different combinations of 'epsilon' and 'max_iter' => trained in nearly a minute, $RMSE=887$, $MAE=166$, it's like it decreased the MAE with 10 but increased it in RMSE... Slightly negative and nearly zero $R^{2}$ too.
  • I have noticed that for linear models the weights have really huge values ($10^{12}) but they compensate each other and the intercept is exactly the mean of the train set. I'd like to point that I have normalized the data first because fitting the linear models, and I'm predicting on normalized validation (using mean and std from train) but I haven't scaled the data for random forests.

Due to the huge amount of time required to train the random forests I haven't tried many combinations of values for the hyperparameters.

What could cause such performances?

According to this post When to avoid Random Forest?, it seems like the two major issues are:

  • Sparsity: Which, even if not measured formally, is observed in the data sets.
  • Data are not axis-aligned: I have no formal way to assess this too but maybe it could explain the long time of training for the random forests as it needed many splits?

I don't know why would all linear models perform the same and just predict the mean while the hubber would change a little bit in favor of improving the $MAE$ but worsening the $RMSE$.


  • I also found that during my cross-validation scheme, I'm not retraining the models from scratch (like retrain the models on [0, T]+[T, 2T] data) but I trained the already-trained models. For neural networks it'll just use the learned weights as initialization but I'm not sure about linear regression and random forests.

Any ideas, hints or clues are highly appreciated. Thank you!

  • $\begingroup$ In lasso, and any time you center all the predictors (which typically happens automatically in most lasso software implementations), the intercept will be exactly equal to the training set mean. That is not a feature of the prediction algorithm but rather a result of the centering process. Have you tried using BART? I find it has good performance out of the box. $\endgroup$
    – Noah
    May 10 at 0:01
  • $\begingroup$ Thank you for your answer @Noah . But that wouldn't explain why the simple Linear Regression as well as Ridge predict the mean too no? The Huber Regressor is slightly different though, but its results doesn't change no matter my hyperparameters combinations. I'll try BART and inform you about its results. I thought maybe in my data set it's hard to find a pattern or there just isn't. $\endgroup$
    – Daviiid
    May 10 at 6:12
  • $\begingroup$ @Noah So I tried the BART implementations I could find on Python but I encountered many issues and thought to give it up for the moment. There's another point I just discovered, during the cross-validation using the Time Seriers Split schema, I'm not retraining the models from scratch do you think that might influence? For Neural Networks this could act as some weight initialization and it could influence the outcome but I don't know about linear models and random forests. $\endgroup$
    – Daviiid
    May 10 at 21:15


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.