Estimating "Probability" in normal probability plot I plotted normal probability plot in R using qqnorm and qqline. I want some help on:


*

*How to estimate "probability" that a data has normal distribution? (I read in a paper that a probability of 0.10 is required to assume that a data is normally distributed). 

*Also, how to calculate correlation coefficient for a normal probability plot in R?

*Is the normality test valid for nonlinear regression too? (This might be a silly question, excuse me for that!)


Thanks!
 A: 
I plotted normal probability plot in R using qqnorm and qqline. 

Great. If you need to assess normality of something, that's a good way to do it.

How to estimate "probability" that a data has normal distribution? 

Unless you want to treat this as a Bayesian problem, you don't. If you're doing a hypothesis test, the p-value is not the probability that the data has a normal distribution.

(I read in a paper that a probability of 0.10 is required to assume that data is normally distributed).

a) Either the paper didn't quite say what you think it did, or it's basically wrong. It might have said something about a p-value of 0.1, but if it says that you need a p-value of 0.1 to assume normality, at best it's an unsupportable generalization that might be suitable for some people in some particular circumstances.
b) Even leaving that aside, you probably don't want to do a formal hypothesis test of normality. (Why are you checking normality?)
Edit: responding to a Q in comments (wording slightly reordered here so it doesn't lose clarity out of context.)

[I'm] testing normality in a nonlinear regression model for microbial growth, because if residuals are not normally distributed, what other distribution or pattern they are following?

The question isn't "are they normal?" (that's essentially never going to be true). The question is more like "does the degree of non-normality we have impact the inference so much we wouldn't be prepared to assume normality?"
-- See this answer starting at "But given that real data..." onward, skipping the two paragraphs starting at "Similar considerations..." but then reading to the end. 

Also, how to calculate correlation coefficient for a normal probability plot in R?

If your data is in y, you'd do it like this: 
 cor(y,qnorm(ppoints(length(y)))[order(order(y))])


Is the normality test valid for nonlinear regression too? 

Almost as valid as it is in linear regression. (Which is to say, I'd generally advise against formal testing of regression assumptions in either case.)
Computing the correlation is fine as far as it goes. If you want to use that in a test, that's already implemented in R (it's the Shapiro-Francia test), the function is sf.test in package nortest. It's nearly as powerful as the Shapiro-Wilk. 
A: Your data are either Normal (in which case the probability that the data have a Normal distribution is 1) or they are not Normal (in which case said probability is 0).  If you'd like to test the null hypothesis that the data are Normal, against the alternative that the data follow some other distribution, you can consider using a Shapiro-Wilk test of Normality.  But the observed significance level (OSL) for that (p-value) is not a probability in the sense you desire: you probably want $P\{$Underlying Normal | Data$\}.$  The OSL is the probability a random sample from a Normal would be that extreme or more extreme.
The choice of a critical value is dependent on the situation.  Ten percent might be reasonable, or it might be too low (or even too high) depending on circumstances.
The Shapiro-Wilk statistic is more-or-less the correlation coefficient you seek.  Strictly speaking, a product-moment correlation coefficient is inappropriate -- by using the order statistics you have induced a correlation.
Again, strictly speaking, tests of Normality are not appropriate for use on residuals because the assumption in the test derivation is i.i.d. observations and ordinary residuals are necessarily correlated.  For large samples, though, the correlation is usually small (of order $n^{-1}$) and generally ignored.  So the tests are as applicable to nonlinear least squares (or maximum likelihood) residuals as they are to linear least squares residuals.
