Rank versus Box-Cox transformation I'm attempting to assess the relationship between two quantitative variables, but the DV is highly skewed (and so are the residuals). I work among biologists who tend to favor non-parametric techniques (e.g., Mann Whitney, Kruskal Wallis). In one part of the paper, a Mann Whitney is used to assess group differences. Although I generally do the Box-Cox when doing simple (or multiple) regression, for continuity I decided to rank transform the DV. Alas, it was all noise (i.e., small effect sizes and non-significant p-values). Out of curiosity, I decided to use a Box-Cox transformation. With that, the p-values became significant and the effect sizes increased.
So, with that background, a couple of questions:


*

*Am I to interpret the discrepancy as due to random fluctuations (i.e., the Box-Cox is committing a Type I error)?

*Am I to interpret the rank transformation as more conservative? (i.e., the rank transformation is committing a type II error). 

*(Related to #1 and #2) Does one transformation tend to be more conservative than the other? Or does it depend on the dataset?
 A: Given that your area tends to favor non-parametric tests, my answer will assume that the ordinal properties of your dependent variable are more important than the interval properties, and so back-transformation is less important to you than is determining whether or not some monotonic relationship exists (please correct if I'm wrong).
Box-Cox transformation could increase Type I error rates if the free parameter  $\lambda$ is selected by maximizing $R^2$, or by some equivalent procedure such as maximizing log-likelihood.  For example, the boxcox() function in R does this by default, and so that could be a concern.  This would be less of a concern if $\lambda$ were chosen solely on the basis of normalizing the dependent variable (without consideration of the IVs), but of course, such a strategy might also lead to less normal residuals.
Considering that the question is about regression, it seems your goal is to get the residuals approximately normally distributed.  For that reason, the answer to all of your questions is "it depends on the dataset."  Although Box-Cox is quite flexible, it won't always be able to address non-normality, and it may sometimes be less useful than a simple rank transformation.  As an extreme example, a Box-Cox transformation would do little to normalize a sample from a Cauchy distribution because of its symmetry and heavy tails, but a rank transformation would at least pull in both tails.  I am not aware of a comparison between Box-Cox and rank-transformation in regression, but you can get some sense by looking at a comparison in correlation.  In the case of correlation, the rank transformation (with the Spearman $r_s$) often leads to smaller Type I and II error rates for symmetric non-normality, but the Box-Cox is sometimes superior for asymmetric, skewed distributions (Bishara & Hittner, 2012).
References:
Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with nonnormal data: comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological methods, 17(3), 399.
