One approach is to use an F-test for goodness of fit (a description is in this question Linear regression: F-test for lack of fit (using ANOVA to test regression model) - intuition?). This requires data that has been sampled with the same conditions (e.g. in lots of scientific fields experimenters perform repetitions of the same experiment under the same circumstances, that gives an expression of the distribution of the error).
Or just use the ratio of the estimates of the error variance. The F-test can be unreliable, too easy with little power, if there are few degrees of freedom, and on the other hand it can be too harsh when a small amount of lack of fit is already significant.
Effectively it is a test to see whether the error terms in the model have zero expectation. You can also do this by visually inspecting the residuals.
In some models the error also relates to the expected value (except if there is potentially over-dispersion, but sometimes that might be excluded based on theoretical grounds). Examples are a binary classification or estimatiom of a Poisson distributed variable. In that case one can compare the observed likelihood or variance with the one that would be expected.