I am missing some understanding here. I am inspecting the relationship between the heart rate variability (HRV) and errors in the Sustained Attention to Response Task. When I conduct a basic linear regression with R with the original values I get close to a significant result. When I inspect my qq-plot I realise that the residuals are not normally distributed (picture 1). Then I decided to do a square root transformation to the HRV data and I find the residuals a bit more normal (picture2). Also, I find a significant result. Here I run into a dilemma. Can I justify transformation if the residuals looks a bit "better" and say I have a significant result, which I do not have with the non-transformed data. This feels a bit wrong. Is it? And what does my results mean now when transformed HRV has a significant association with the performance in this test? The HRV is, of course, a somewhat arbitrary number and the participants still stay comparable in that matter even though I do the transformation, but somehow the relationship is changed. And did the transformation made residuals normal enough that I can do linear regression? Cheers!
As Mike Hunter notes in a comment, neither of your models is doing a very good job of providing residual distributions that are close to normal. There are two issues here: correctly modeling the shape of the relationship between predictor and outcome (which we don't see described in this question), and modeling the residuals of observations about the predictions. Results like yours could be due to difficulties with either of these.
For the first task, one of the best approaches with a continuous predictor is to model it as a restricted cubic spline rather than as a single polynomial. You choose a small number of knots. The regression itself then fits a smooth curve through those knots, and standard methods can determine whether adding additional knots significantly improves the fit. Thus you don't have to guess the correct form to start, and you have a principled way to choose the flexibility of the fit. This regression approach is provided, for example, by the
rcs() function in the R
Even if you fit the proper general form, however, your residuals about the fit might not be normally distributed. That depends in part on the nature of the outcome variable. If your outcome variable is the number of errors in a task, it might be impossible to have normally distributed errors if the outcome isn't close enough to continuous. Count data are sometimes better fit with a generalized linear model that explicitly models the counts with Poisson or negative-binomial distributions. Combined with proper modeling of the overall shape of the relationship, you then can obtain estimates of statistical significance that are based on better approximations than the normal to the underlying error distributions.