Prove that Residuals/Errors are not Predictable [duplicate]

Question

Let's say I have a set of data and I modelled it using an ML algorithm. After I fit multiple models I achieve a certain level of accuracy. It doesn't get better beyond that point.

I want to prove using the error term that there are evident patterns and since there is predictability in error term therefore I can't model better than that.

One way to do that would be to visualize the error terms and see if there are any evident patterns or not. But a more robust way to that would be by using a statistical test.

Which statistical test can prove or disprove any evidence of patterns in residual?

Answer 1

One approach is to use an F-test for goodness of fit (a description is in this question Linear regression: F-test for lack of fit (using ANOVA to test regression model) - intuition?). This requires data that has been sampled with the same conditions (e.g. in lots of scientific fields experimenters perform repetitions of the same experiment under the same circumstances, that gives an expression of the distribution of the error).

^{Or just use the ratio of the estimates of the error variance. The F-test can be unreliable, too easy with little power, if there are few degrees of freedom, and on the other hand it can be too harsh when a small amount of lack of fit is already significant.}

Effectively it is a test to see whether the error terms in the model have zero expectation. You can also do this by visually inspecting the residuals.

In some models the error also relates to the expected value (except if there is potentially over-dispersion, but sometimes that might be excluded based on theoretical grounds). Examples are a binary classification or estimatiom of a Poisson distributed variable. In that case one can compare the observed likelihood or variance with the one that would be expected.