1
$\begingroup$

I have an experiment with N=34 measured points. For each point I measure a variable $x$ and a variable $y$. In my paper I reported the average and the standard deviation of $x_i -y_i$ (where $i$ is the sample index). I've qualitatively plotted the scatter plot $x-y$ vs $x$ and I've stated that I do not see any specific trend for $x-y$ vs $x$. Now the reviewer wants me to perform "a statistical test" (without any other specification) to exclude that any such trend exists. How can I do that?

If I measure the correlation coefficient between $x-y$ and $x$ I do find $r \approx 0.45$ which is statistically significant at p<0.05 with N = 34 points. However I'm pretty sure it's due to 4 points that are a bit outside of the normal distribution tails. If I remove them then $r\approx0.1$! These 4 points are clearly outside the main cloud of points.

So, what kind of test can I use to exclude something? Should I measure the average $r$ of a bootstrapping experiment? Is there any other test?

Edit: I've added the scatter plot in question. enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ If you want an advice for your specific case then consider showing here your scatter plot. $\endgroup$
    – amoeba
    Jun 11, 2018 at 9:29
  • 1
    $\begingroup$ I've added the plot. Anyway, I'd be happy even with the concept in general about how to exclude that something is present, without a clear hypothesis about what might be present to begin with. $\endgroup$
    – user26067
    Jun 11, 2018 at 9:35
  • $\begingroup$ (1) Are you sure $r=0.45$ and not $-0.45$? (2) An alternative impression is that $x-y$ varies linearly with $x$, up to some random error. In that case there are no outlying points--but there is a trend. On the face of it--when there is no underlying theory as a guide--this interpretation is the superior one, because it is substantially simpler: describing a trend requires just one additional parameter (the slope) whereas excluding four points requires four additional parameters. (3) You can't test this with the data, because your four-outlier model was developed by examining the same data. $\endgroup$
    – whuber ♦
    Jun 11, 2018 at 13:42

1 Answer 1

Reset to default
1
$\begingroup$

What is asked by the reviewer is not super clear for me...

What I would do is to run a method for detecting outliers. One solution for that can be to use cook distances, which define how influential points are on a model. The cook distance of a point i is the difference of model output with the point i and without.

Improve this answer
$\endgroup$
2
  • $\begingroup$ For the pure "correlation between y-x an x" question you can use Spearman's $\rho$ to get a robust correlation measure and test. But I don't think I would have opened this up to reviewer scrutiny. I would have used loess to get a smooth nonparametric estimate between x and y. Also compute mean |x - y|. $\endgroup$
    – Frank Harrell
    Jun 11, 2018 at 10:46
  • $\begingroup$ Yeah, but at this time I don't have a model! I'll give spearman's a try! $\endgroup$
    – user26067
    Jun 11, 2018 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.