@TheGreat the link to your question doesn't work anymore! what solution are you looking for? this post addresses several issues i think.

oh, and another remark: the feature weights (no matter with or without intercept) do not have to sum up to the prediction.

Hey! I am also looking into ways of how to understand the intercept in LIME. My intuition/understanding is (by looking at many explanations of many different models), that the intercept shows how much cannot be explained by feature weights. Does that make sense? Did you find another interpretation of the intercept term?

@MarcoFumagalli by just looking at the similarity/difference between $V$ and $WH$ you will choose a high $k$ because the higher the $k$ the lower the error will be (mentioned in the second answer in the paragraph RSS against randomized data).

That's a good start thanks! Any ideas where to get a dataset with more observations?

Simulation is always good but if you want to publish an algorithm you want to have comparable results to others.

is it possible to use PRESS just as a validation method for a recursive least squares solution? I'm not even sure anymore if I can use it just as a validation method without using it for parameter selection. My plan is: Use recursive least squares (in a simplified scheme with lambda = 1) to estimate parameters. use PRESS on the output to validate the model. Is this possible or completely nonsense? @DikranMarsupial

Did you have any luck?