What really happens when we transform the data using $f(x) = \sin(\sqrt{x})$? I need to perform a two-way ANOVA on my data ($Y$: sleeping hours). My data is quite normal $p$-value = $0.07$ with Shapiro-Wilk test but when I run the normality test for my residual, $p$-value is less than $0.01$. So I did the transformation by using $f(x) = \sin(\sqrt{x})$. My new normality test for residual from coded data is 0.03. My question is


*

*Is $\alpha = 0.03$ enough for the normality test to perform two-way ANOVA? I have tried $\ln(x)$, $\sin(x)$, inverse, $\sqrt{x}$, but they are all worse.

*What does $f(x) = \sin(\sqrt{x})$ really do to my data? 




 A: Nick Cox has done a good job talking about the transformation of your data. Let me address some other issues in your question.  
You state that your data are "quite normal" because the $p$-value from a Shapiro-Wilk test is $.07$.  There are several problems here.  First, the distribution of your data is irrelevant; only the distribution of the residuals is important.  (To understand this more fully, see: What if the residuals are normally distributed, but y is not?)  Second, there is no 'bright line' at $.05$.  $.07$ is pretty close to $.05$, and unless you powered your study to reject normality at a specific level of $W$ for some reason, I would say that is enough to be called 'significant'.  (For more on this, see my answer here: When to use Fisher & Neyman-Pearson framework?, also this quote.)  Moreover, the normal distribution is just a mathematical idealization, no data are ever really normal (quote).  In particular, your data can't be $<0$ and shows evidence of rounding; with enough data, there is no question you would get a $p$-value $<.05$ (cf.: Is normality testing 'essentially useless'?).  
A: I still don't understand exactly what you did, but here is a plot of sin(sqrt(hours of sleep)) over the approximate range observed and between the absolute limits of 0 and 24 hours. 
By accident the transformation is monotonic for the observed range, although almost linear, so the effect on the distribution should be slight. 
It is quite absurd otherwise. It has no basis theoretically. 
In terms of the bigger question, it's not clear to me that the distribution of the data is awkward for ANOVA at all.  
EDIT Small print: Your first graph I read as showing data from about 3 to about 11 hours, you say you have 4 to 10 and transformed you have 0.7 to -0.02. But although sin(root(10) is about -0.02, sin(root(4)) is about 0.91. So there are small contradictions here. Regardless of that, this transformation is indefensible, as it is not even monotonic over the possible range. 

