I am making my first thorough exploration of transformations.
My primary goal is to improve the normality of the residuals following a mixed effects model fit. For some of my response variables, the residuals (and original data) show a good approximation of a normal distribution. For others, there is substantial skew. I have tried several standard transformations (log, sqrt, squared, etc.) and for most response variables one of these works well.
I know there are arguments for and against response variable transformations, but at least some things I have read indicate that this empirical approach is acceptable.
Assuming that is indeed acceptable, my question is whether it is also acceptable to choose an arbitrary power transformation. One set of typical transformations include raising the response variable to some integer power: -3, -2, -1, 1, 2 etc. (cube root, through reciprocal, no transform, and squared). In one of my response variables, none of these provides a good approximation of normality. Instead, I found that raising the response variable to a fractional power (0.25) worked quite well. Theoretically, one could also find that raising to the power 1.5 works better than no transformation or the squared transformation.
I have not been able to find any discussion of whether this is a good or bad idea.
So, to summarize: assuming that empirical selection of a transformation to improve normality is acceptable, should the power transformations explored be limited to integer powers or can any fractional power be considered?