Probability that residuals are normal I want to make a quantitative statement like "There is a 90% chance that this $X$-$Y$-data follows a linear model (with some noise added on top)". I can't find this kind of statement discussed in standard statistics texbooks, such as James et al.'s "An Introduction to Statistical Learning" (asking as a physicist with rudimentary statistics knowledge).
To be more precise: I'm assuming that some data is generated from $Y = f(X) + \epsilon$, where $f(X)$ is some exact relationship, e.g. the linear model $f(X) = \beta_0 + \beta_1 X$, and $\epsilon$ is noise drawn from a normal distribution with some unknown standard deviation $\sigma$. I want to calculate the probability that some proposed $\hat f(X)$ matches the actual $f(X)$.
I can do a least-squares fit to determine the estimate $\hat f(X)$. Now, if the model is correct ($\hat f(X) = f(X)$), then the residuals of the fit should exactly correspond to $\epsilon$. At the very least, if the data fits the model, there should be no correlation between the residuals and $X$. To be more quantitative, though, I would want to check that the residuals are in fact from a normal distribution with unkown $\sigma$ (although the residual standard error, RSE, will be an estimate for $\sigma$, so I could also assume that $\sigma$ is actually known). Isn't there some way to calculate a p-value for whether some given values (the residuals) are from a given distribution (normal distribution with RSE as the standard deviation)?
I'm not looking for the $R^2$ statistic, which will tell me how linear the data is, but also take into account the noise (larger $\sigma$ will lower the $R^2$ value). In my case, I don't care how noisy the data is, as long as it's normally distributed around the fit $\hat f(X)$.
 A: One difficulty in presenting a single number as you describe, is that the statistical confidence in the fitted model - here a straight line - can be different at the low or high end of the data range. A single number does not capture this, which you can see here in these animations of 95% confidence intervals for different curve fitting problems:
http://zunzun.com/CommonProblems/
A: QQ Plot


*

*For comparing if the residuals are coming from normal distribution, you can use a QQ plot. 

*On the X-axis are the Theoretical quantile and Y axis are the distribution obtained from the data set. If the points in the figure lie on the straight line, then the model is useful and satisfied the normality assumption of linear regression.



KL divergence (Kulbeck Leibler divergence)


*

*For comparing 2 normal distributions with $N_1 = \mu_1, \sigma_1$ and $N_2 = \mu_2, \sigma_2$

*$KL(N_2, N_1) = \log{\frac{\sigma_2}{\sigma_1}} - \frac{1}{2} + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^{2}}$ (lower means more match between 2 distributions)
If the 2 distributions are the same, then the KL will be zero.
For your use case, put $\mu_1 = 0$, $\mu_2 = 0$, and $\sigma$ as you wish.
For p-value


*

*Check out Bonnet's Test or Levene's Test
