# In R, how can I transform to normalize residuals when I have a U-shaped Q-Q plot?

I am running a two-way ANOVA with one random variable. My histogram of the residuals is showing considerable (negative?) skew:

And my Q-Q plot of the residuals shows a corresponding U-shaped curve:

Can someone suggest an appropriate transformation to normalize these residuals? I have tried log and sqrt transform, but these aren't doing the trick - Q-Q plots look essentially identical.

Here's the residuals just in case anyone needs them:

resids<-c(0.0907115234404372, -0.00228847655956277, -0.0267884765595627,
-0.0859262641306289, -0.0212262641306287, -0.0667262641306288,
-0.0593952037730363, 0.0185047962269638, -0.0364802818046719,
-0.0345802818046719, -0.0636802818046718, 0.0459197181953281,
0.110319718195328, 0.00411971819532808, 0.0156218781924411, 0.0877218781924409,
-0.0467325370980458, -0.0373134598697733, 0.0841865401302266,
-0.0560134598697732, 0.150586540130227, -0.0111134598697733,
0.152686540130227, -0.045845932936565, -0.0389459329365651, -0.0163459329365649,
-0.00584593293656499, -0.015545932936565, 0.00425406706343501,
0.0194540670634349, -0.0566459329365649, -0.0370459329365651,
0.0271762555042805, -0.0518237444957197, -0.0459770706142297,
-0.0256237444957197, 0.105910766841122, 0.0239136558615476, 0.0456136558615476,
-0.0437396702569626, -0.00593967025696251, -0.0071396702569626,
-0.0236257538123323, 0.0136019650946468, -0.0111257538123324,
-0.00702575381233239, -0.0239257538123323, -0.0280284968467248,
0.0705715031532752, -0.0504829121372117, -0.0429829121372116,
-0.0193829121372118, -0.0509829121372116, 0.10563489248827, -0.0311651075117305,
-0.0259651075117306, 0.0650348924882695, 0.0251893077787564,
-0.00731069222124359, -0.0845911986230001, 0.0188349345620156,
-0.00856506543798452, -0.0402650654379846, -0.00956506543798441,
-0.0695650654379845, 0.0615349345620155, -0.0495650654379844,
0.0476936955538023, -0.0414063044461976, 0.0333936955538023,
-0.0451063044461977, -0.0531413637369638, -0.045694689855474,
0.000385664780562678, 0.0940389908990729, -0.0195610091009271,
-0.0254143352194374, -0.0284143352194373, 0.000298786199770751,
-0.0942012138002293, -0.0013012138002293, 0.00800940250009097,
-0.0193946496405897, 0.0119330692663895, 0.0370053503594103,
-0.0113669307336104, -0.0499459456813103, -0.0415459456813103,
-0.0229459456813104, -0.0300459456813102, 0.0462540543186898,
0.0443540543186898, 0.0407540543186897, 0.0183540543186897, 0.0571540543186897,
0.0363632617174681, -0.0610367382825319, 0.0981632617174681,
0.0474632617174682, -0.0485367382825319, -0.0585367382825319,
0.00866326171746823, 0.0131632617174682, 0.190063261717468, -0.0370367382825318,
0.0362590810548065, 0.0105590810548064, -0.0385409189451935,
0.0369590810548064, 0.0492590810548064, -0.0298409189451936,
-0.0589915546215696, -0.00219155462156961, -0.0737915546215695,
0.0181084453784304, -0.0104915546215696, 0.0627084453784306,
-0.0674915546215695, -0.0622915546215694, -0.0274840518137136,
-0.0470117707206927, 0.0133159481862863, 0.0356882292793073,
-0.0581584446021826, -0.0386584446021827, 0.0445783565165312,
-0.0369216434834687, -0.00882164348346892, 0.0411250303980211,
-0.0485216434834688, -0.021339528891233, 0.0656387579342828,
-0.0313612420657172, -0.023839528891233, -0.0142395288912331,
-0.029639528891233, -0.0310499966187923, 0.0433500033812078,
0.0617500033812077, -0.0366499966187923, -0.0621499966187924,
0.0211500033812078, -0.0632499966187923, 0.0631500033812076,
0.0819500033812077, -0.000349996618792314, -0.0129198942139448,
-0.0450198942139448, 0.130180105786055, 0.138980105786055, -0.0668198942139449,
-0.065919894213945, -0.0167652600956127, -0.00446526009561277,
0.0310347399043871, 0.0220564530788714, 0.0179564530788714, 0.0407347399043871,
0.0111347399043871, 0.0776564530788715, -0.00805388766299564,
-0.0662538876629957, -0.0307083029534827, 0.0541461123370044,
0.113446112337004, -0.0603320519387343, -0.0217320519387341,
-0.0915320519387341, 0.113074320307902, -0.0297256796920982,
0.0209743203079018, -0.0236256796920982, 0.0587743203079019,
-0.0662256796920981, -0.0534256796920982, -0.0338256796920982,
-0.083325679692098, 0.00277707299336427, -0.0294229270066357,
0.0124770729933641, -0.0533229270066358, 0.146077072993364, 0.0317770729933642,
-0.0685448001543059, -0.0316448001543057, -0.0575448001543057,
0.146509615136181, 0.014109615136181, -0.0747903848638189, -0.0166777624015848,
0.0308222375984151, 0.0914222375984151, 0.00242223759841509,
-0.0389777624015848, -0.000977762401584981, 0.0269222375984151,
-0.0204624068697834, 0.102637593130217, -0.0698624068697833,
-0.0742624068697832, -0.0477624068697833, -0.00586240686978323,
0.0955375931302167, 0.0890642743432923, 0.180464274343292, -0.044757438831192,
-0.0723357256567077, -0.0680357256567077, -0.00925287584284296,
-0.0209072911333299, -0.0347528758428428, 0.204647124157157,
0.113207730323683, 0.00190773032368274, -0.0242922696763173,
-0.0405922696763172, -0.0265922696763172, 0.0974077303236829,
-0.0243922696763172, -0.0630922696763172, 0.0283077303236827,
0.0582681073945803, -0.0279318926054197, -0.0236318926054198,
-0.0272318926054196, -0.0347318926054196, -0.0801318926054198,
-0.0626768408136609, 0.0253502995705317, -0.0571497004294683,
0.200850299570532, -0.0440497004294684, -0.0357497004294682,
0.0217502995705317, -0.0505861289667355, 0.0888138710332644,
-0.0749861289667355, 0.0619138710332645, 0.0599138710332645,
-0.0103861289667355, -0.0159861289667356, 0.00541387103326452,
-0.0678160116680873, 0.0365839883319128, 0.0577839883319127,
0.0136839883319126, 0.0878839883319127, 0.0160839883319128, 0.144683988331913,
-0.0139160116680872, -0.0400766221886237, -0.0801766221886238,
-0.0895766221886236, -0.00807662218862371, 0.0238233778113763,
-0.0527766221886237, -0.0292766221886236, 0.00982337781137632,
-0.00237662218862367, -0.0566766221886237, 0.0823492874218354,
0.0203948721313485, -0.0382507125781646, -0.110305127868652,
-0.0752051278686516, 0.153894872131348, 0.100494872131349, 0.0305948721313485,
-0.0123051278686515, -0.0669051278686517, -0.0574913489020565,
0.107108651097943, 0.0480086510979434, 0.0452086510979433, -0.101691348902057,
0.00680865109794326, 0.0692086510979433, -0.0405913489020566,
0.0349086510979433, 0.0499356114062059, -0.0297861017682783,
-0.0142861017682785, 0.0234138982317216, 0.0147356114062058,
0.0696356114062058, -0.0285643885937941, 0.0578818302164203,
-0.0486181697835797, -0.0167181697835797, 0.0221818302164203,
-0.0665338718584472, 0.00596612814155284, 0.0139661281415528,
-0.0721338718584472, -0.0639338718584472, 0.119566128141553,
-0.0489338718584471, -0.0740338718584472, 0.00296612814155295,
-0.0787409112120181, -0.0295409112120182, -0.0417409112120182,
0.0827531906259826, -0.0251468093740175, 0.0166531906259826,
0.00825319062598262, -0.0569468093740175, -0.0354468093740175,
0.0578531906259825, 0.0127531906259826, -0.0787468093740173)

-

The boxcox function in the MASS package will give an appropriate range of $\lambda$ values for the Box-Cox transformation. If you have not read the paper by Box and Cox, then you should.

Combine the suggested range with knowledge about the science that generated the data and some common sense to decide on a final value (don't just use $\lambda=0.413$ because that gives the best answer, if the confidence interval includes $0.333$ and $0.5$ then look to see if a square root or cube root makes sense with the science and use the one that makes the most sense).

-

The log transformed data looks not bad:

trans <- log10(resids-1.0001*min(resids))
qqnorm(trans)
qqline(trans)


If that is appropriate depends on the science behind the data.

-
Thanks Roland, but as I understand it, I need to perform the transformation on the response variable rather than on the residuals. Unfortunately, log transformation on the response variable (even the variant that you used) still results in a U-shaped curve of the residuals in the Q-Q plot. BTW, what is the -1.0001*min(resids) part of your log transformation intended to do? – Luke Sep 26 '12 at 14:47
@Luke That was not clear to me from your question. You might want to look at lme4::lmer or nlme::lme and their family argument. Substracting a bit more than the min avoids negative or zero values when taking the log. – Roland Sep 26 '12 at 15:24