I am running a random forest on a dataset (only numerical) and achieve the following predictions:

Predictons vs truth values

Thanks to question and answers such as here, I understand why the predictions vs truth values don't lie around the y ~ x line.

I could stop here and accept this as the final model but I have a couple of questions around potentially improving it.

  1. Is it reasonable to fit a linear model to the predictions vs truth to correct for the mismatch between these and the y ~ x line? Instinctively I think this is probably a bad idea, but I'm not entirely sure why. I've seen a blog that starts with a linear model and then fits an RF to the residuals - can't find it again now, sorry (Edit: found it) - perhaps that's a better way to go?
  2. How best to deal with the heteroscedastic errors? I have already done all the transformations that seem reasonable - e.g. % variables to logit, log-normal for strictly positive variables near zero, obvious things like that. I don't think I need to worry about including interaction terms as I thought RFs would handle those automatically. Maybe there's something obvious I'm missing but, failing that, what's the best next step? I could combine this with question 1 using a glm I suppose?

I'm really at the limits of my modelling skills now and would really appreciate some pointers.

Edit: just to clarify - I care about improving the predictions not whether the errors are heteroscedastic or not per se. I'm just hoping if I can improve that then I can improve the noise, particularly at the low values, and get better predictions. But as @Tim noted, it probably more an issue of data quantity in that region than anything as there's no reason why reducing the heteroscedasticity ought to improve the predictions. That still leaves question 1 though - is it reasonable to regress on the fit?

  • 1
    $\begingroup$ Why do you care about heteroscedasticity of the errors? $\endgroup$
    – EdM
    May 18, 2021 at 11:29
  • $\begingroup$ Partly because I feel like I ought to! But also because I wonder if I can address that I can reduce the noise in the predictions at the lower values. Edit: just to add - these are the most difficult tests to achieve for the technicians (which is probably part of the reason why more noise - but also some technical reasons why I suspect) so it's actually the lower end that has the most impact of being able to predict well. $\endgroup$
    – Mooks
    May 18, 2021 at 11:34

1 Answer 1

  • Start with the Why don't linear regression assumptions matter in machine learning? thread. Assumptions for statistical models are needed for inference, while with machine learning models we care about making accurate predictions.
  • Even in the case of linear regression, there are misconceptions about its assumptions and their importance, as you can learn from the What is a complete list of the usual assumptions for linear regression? thread. Homoskedasticity in linear regression is needed for calculating confidence intervals and running hypothesis tests, not the things you care about or even are possible to do with random forest. If you don't care about those, you can even use linear regression if the assumption is not met.
  • The homoskedasticity assumption comes from the distributional assumption about independent and identically distributed noise $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$. The random forest does not make any distributional assumptions.

So nothing is broken and nothing needs fixing. You may care about the predictions being biased and may want to do things like gathering more data (especially for the values <60 and >70), feature engineering, hyperparameter tuning, ensembling models, etc, to improve the predictions, but it does not have anything to do with homoskedasticity.

  • $\begingroup$ Hello Tim. Thanks for this. I do appreciate the over enthusiasm for people to care about things like multicollinearity and heteroscedastic errors in ML when all you care about is prediction. Maybe my question wasn't very clear, sorry. I am wondering if there's anything I can do about the bias and/or non-normal errors to improve the predictions? I've done some basic feature engineering (transformations) but kind of stuck with what else to do. Tuning doesn't seem to improve the situation much. I've also seen it in auto ML with H2O and others. $\endgroup$
    – Mooks
    May 18, 2021 at 12:03
  • $\begingroup$ @Mooks you have no reason to care about the normality of errors. As said above, you can do the standard machine learning things to improve the model. I would mostly recommend gathering more data from the ranges where the predictions are worst since the region with most data seems to be predicted quite well. $\endgroup$
    – Tim
    May 18, 2021 at 12:05
  • $\begingroup$ Ah ok. I was half hoping that improving the normality of the errors might tighten the noise and/or bias. I suppose that’s more hope than judgement! I hadn’t really thought about the difference in data amounts - thanks for pointing that out. Alas this is historic data over 30 - 40 years so I probably won’t be able to gain more, at least not quickly! I’ll carry on fiddling with the standard ML stuff then. Thank you. $\endgroup$
    – Mooks
    May 18, 2021 at 12:10
  • $\begingroup$ If you have heteroscedasticity, you can get improved predictions by weighting. Specifically, in pockets where the variance is small, you can completely miss the entire pocket on one side or the other if you do not weight. $\endgroup$ May 18, 2021 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.