5
$\begingroup$

Statistics beginner here so please don't shoot if the terminology isn't 100% neat.

In a sample of 45 individuals, I am looking at a relationship between two parameters (Annual_SOM_rate and Share_temp_past). When I calculate the p-value of the slope (linear regression), the result is significant: enter image description here

However, the significance disappears when I omit the value in the top right corner (green circle). Although a little bit "alone", this value makes total sense from a practical point of view (we cannot discard it as an error). If I extend this argument, would removing other value also impact the relationship?

My question: which statistical operation would help me conclude if the relationship that I observe here is truly significant or not?

Would boostraping help me ?

$\endgroup$
1
  • $\begingroup$ You may be interested in the "fragility index", which measures how many values must change to reverse significance. $\endgroup$
    – Ben
    2 days ago

1 Answer 1

7
$\begingroup$

First off:

If I extend this argument, would removing other value also impact the relationship?

Almost certainly yes. The regression line slope depends on all observations, so removing any point will change the estimate, and therefore also the $p$ value.

(It is possible, but unlikely, that there are data points with no influence on the coefficient estimate. Removing such a point will not change the parameter estimate, but will change the $p$ value slightly, since this depends on the parameter estimate and the number of observations.)

Removing a data point may or may not move your $p$ value above or below the magic threshold of $p < .05$.

which statistical operation would help me conclude if the relationship that I observe here is truly significant or not?

Statistical significance depends on the observed data (and a correctly specified model). There is no "true" significance. Either the null hypothesis holds (e.g., there is no relationship), or it does not (e.g., there is a relationship) - and many statisticians will say that the null hypothesis never holds in the first place. All we can say given data (and a model) is whether the data are consistent with a null hypothesis or not.

Thus, if you have no reason to suspect a data error, you should take your $p$ value as it is. It would be good to be cautious about any conclusions you draw and note explicitly that significance depends on a single data point. Also, note that the difference between the slopes you estimate with and without this one data point will probably in itself not be significant (Gelman & Stern, 2006).

Would boostraping help me ?

Bootstrapping is just another way of looking at given data. It will give slightly different results. You may get $p<.05$ with the full dataset with a bootstrap - or not, and the same if you omit this observation. (And as above, a bootstrapped and a parametric parameter estimate may not be significantly different.) If all you are interested in is significance, then I would not worry overly about differences between a parametric and a bootstrap analysis.

Edge cases like these are one main reason to be careful not to over-interpret statistical significance. If you want more precision in your estimates, there is no way around collecting (a lot) more data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.