I plotted normal probability plot in R using qqnorm and qqline.
Great. If you need to assess normality of something, that's a good way to do it.
How to estimate "probability" that a data has normal distribution?
Unless you want to treat this as a Bayesian problem, you don't. If you're doing a hypothesis test, the p-value is not the probability that the data has a normal distribution.
(I read in a paper that a probability of 0.10 is required to assume that data is normally distributed).
a) Either the paper didn't quite say what you think it did, or it's basically wrong. It might have said something about a p-value of 0.1, but if it says that you need a p-value of 0.1 to assume normality, at best it's an unsupportable generalization that might be suitable for some people in some particular circumstances.
b) Even leaving that aside, you probably don't want to do a formal hypothesis test of normality. (Why are you checking normality?)
Edit: responding to a Q in comments (wording slightly reordered here so it doesn't lose clarity out of context.)
[I'm] testing normality in a nonlinear regression model for microbial growth, because if residuals are not normally distributed, what other distribution or pattern they are following?
The question isn't "are they normal?" (that's essentially never going to be true). The question is more like "does the degree of non-normality we have impact the inference so much we wouldn't be prepared to assume normality?"
-- See this answer starting at "But given that real data..." onward, skipping the two paragraphs starting at "Similar considerations..." but then reading to the end.
Also, how to calculate correlation coefficient for a normal probability plot in R?
If your data is in
y, you'd do it like this:
Is the normality test valid for nonlinear regression too?
Almost as valid as it is in linear regression. (Which is to say, I'd generally advise against formal testing of regression assumptions in either case.)
Computing the correlation is fine as far as it goes. If you want to use that in a test, that's already implemented in R (it's the Shapiro-Francia test), the function is
sf.test in package
nortest. It's nearly as powerful as the Shapiro-Wilk.