1
$\begingroup$

enter image description here

I am currently building a house price predictor. My lightgbm errors look like the one the one below (illustrative). It shows that there is a pattern in my errors. Can someone explain how to resolve this? My model currently appears to be predicting closer to the mean value. Does this mean my features don't have enough resolve?

error = actual - predicted

Thanks in advance

$\endgroup$
0
$\begingroup$

When we have pronounced patterns in model residuals it makes sense to try and identify if they correspond to a particular variable (or set of variables) in our data. It might be due to multiple reasons. Some common ones are: 1. Omitted-variable_bias, 2. data leakage or 3. covariate/domain shift. Finally it might also mean that 4. the model at hand is quite weak and is only able to recognise some very basic clusters but it otherwise predicts a mean. In any case, if you have strong obvious patterns on the model residuals I would strong recommend trying cluster them, colour code them and try to see if cluster-label is associated with any of the existing features.

A bit more details on the potential reasons I mentioned:

  • During omitted variable bias we might unintentionally ignore a obvious hierarchy or clustering of the data (e.g. customers' countries). This is actually leads to fairly straight-forward clusters on the residual plots. Especially when dealing with mixed-effects models, this is an obvious things too check but it might be present when working with GBMs too.
  • When having data leakage and/or domain shift we might be predicting against a test-sample where part of it is a different distribution from the one we originally trained. In the former case part of our sample contains information that is wrongly included as an explanatory feature but allows us to have an unrealistically strong performance. This creates blobs of "high" performance and "low" performance. Similarly this can happen due to domain shift but in this case we don't wrongly include some information; just our part of our test sample is not representative of our training sample (and thus we get the blobs of varying performances).
  • If we have a weakly performing model, which is barely able to find some association in our data, we might see these kind of parallel string-like tracks. These tracks correspond to some feature x that might be somewhat informative but on its own is it unable to differentiate a lot our general sample. The model ends up effectively predicting the group average along each level of feature x. This differentiates the predictions a bit but within a particular level of feature x the prediction is pretty stable, this then leading to the error within level of feature x appearing at tracks along the slope of the overall response variable levels.

Looking at the plot I think that ether omitted-variable bias or poor performance are the most likely culprits. As mentioned, do colour-code your clusters and perform some basic EDA to associate clusters with existing features before trying more sophisticated residual analysis techniques.

$\endgroup$
  • $\begingroup$ Thank you for the detailed answer!. Sorry I could not respond sooner $\endgroup$ – user13874 Dec 27 '20 at 17:11
  • $\begingroup$ I am glad I could help! Happy (upcoming) New Year! :) $\endgroup$ – usεr11852 Dec 27 '20 at 18:20
  • $\begingroup$ Have a great 2021 :) $\endgroup$ – user13874 Dec 27 '20 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.