2
$\begingroup$

Consider the following small dataset (around 569 data points), where Uptake is the regression target:

Dataset Hist

As you can see, most of the variables are skewed, with some of them having only 2 or 3 data points in some regions (e.g. VT).

To get some baselines, I decided to run H2O's Random Forest AutoML on different splits of this dataset, as well as on a reduced dataset where the "outliers" (they're not noise, just values in regions where observations are scarce) were removed (about 300 data points remained).

On the original dataset, some splits had very good $R^2$ (close to 0.95)on the test set, but most had average MSE (~3.2). On the other hand, on the reduced dataset, most splits had very good MSE on the test set, but the best $R^2$ across all splits was something around 0.65.

Can anyone explain why this might be happening? Are these "outliers" messing with the $R^2$?

Also, any advice on how these features should be handled (e.g. log transform, ...)?

Improve this question
$\endgroup$
4
  • $\begingroup$ Did you consider that by removing the outliers you made the default mean model better. This will make it harder to beat (your R^2 decreases) while your MSE also goes down. It shows you that on this easier dataset you do not outperform the mean estimator that much anymore. $\endgroup$
    – Ggjj11
    Commented Sep 4 at 6:56
  • 2
    $\begingroup$ A well-placed outlier can bring the R-squared close to 1. $\endgroup$
    – Michael M
    Commented Sep 4 at 7:03
  • $\begingroup$ Uptake and most of the predictors are log-normally distributed. So you might try a skew-zero transform, like using ranks, percentiles, or van der Waerden scores (VDW). VDW scores are nothing more than inputting percentiles ($pct$) into the inverse cumulative normal function to obtain a Z-score, i.e., $Z=\boldsymbol{\Phi}^{-1}(pct)$. Histograms of VDWs for all of your variables will reveal purely standard normal variates, N(0,1). Try VDW as a way to remove outlier effects. $\endgroup$
    – wjktrs
    Commented Sep 4 at 13:11
  • $\begingroup$ What regression method are you using? Beware that $R^2$ is normalized only for ordinary least squares. $\endgroup$
    – cdalitz
    Commented Sep 4 at 13:29

1 Answer 1

Reset to default
4
$\begingroup$

  1. The simple answer here is that you are comparing performance on two different datasets. There's no basis for saying one metric goes up or down when the dataset (and range of response values) itself has changed by your decision to selectively remove points. As Michael M says, even deleting one 'outlier' can change model performance metrics, and you have deleted nearly half the dataset - it is not surprising that they change substantially.

  2. Deleting 'outliers' is almost never a good idea unless you are confident they represent errors. See Is it OK to remove outliers from data? and Outlier detection for skewed data

  3. There is insufficient information here for us to offer good advice on a transformation. That decision often benefits from a careful consideration of the problem, data, and intended use of the model results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.