The phenomenon you observe is caused by the use of Spearman's $\rho$ coefficients, which has the undesirable property that it is easily swayed by the presence in the sample of even a few observations that happen to depart from the bivariate ellipse fitting of the bulk of the data.
The general solution to this problem is to replace the Spearman $\rho$ by a robust alternative. In a nutshell, robust estimation procedures were designed to fit a model that is cannot be easily swayed by a handful of non-conforming data-points.
A popular such algorithm is FastMCD [1,2]. The aim of FMCD is to find the model that best fits the central observations in your dataset. Observations that are far from the FMCD fit are flagged as outliers. They do not part take in the estimation of the coefficients fitted by FMCD. They should be modelled separately from the remaining bulk of the data: their very distance from the FMCD fit makes them unduly influential on any classical statistical analysis that would include them alongside the rest of the data-set.
Now, I turn to your specific problem. Here is the code to run FMCD on your dataset:
X <- c(6, 51, 111, 123, 157, 168, 195, 195, 200, 205, 214, 222, 229, 234, 241, 256,
314, 340, 352, 368, 427, 431, 442, 473, 501, 523, 994)
Y <- c(2, 16, 12, 41, 28, 11, 36, 13, 16, 17, 15, 18, 19, 33, 75, 12,
75, 22, 50, 5, 26, 45, 14, 9, 27, 9, 98)
library("rrcov")
library("fBasics")
a1 <- CovMcd(cbind(X, Y), nsamp="exact")
#Spearman rho on the observation not classified as outliers:
spearmanTest(X[[email protected]==1], Y[[email protected]==1])
These observations row index: (2,3,6,8,9,10,11,12,13,16,18,20,23,24,26) are, in the sense of FMCD, those most central point in your dataset. Furthermore, the following observations: (4,7,15,17,19,22,27) have been identified by FMCD to be highly influential: they stand so far out from the model fitting the bulk of the data that they, if left unchecked, will (either alone, or together) single handedly sway the resulting fit. Finally, the Spearman $\rho$ of the observations not flagged as outliers by FMCD is now $\approx0.07$ (p=0.77).
In short, the main advantage of the FMCD fit lies in its resilience (the technical term is robustness) to the contamination of the sample by a few inconsistent data points. For example, it would be very difficult to find 4 points such that by removing them will significantly alter the solution found by FMCD.
EDIT:
As suggested by @amoeba I'm illustrating the resilience of the FMCD to the removal of 4 data points from the set (those with indexes 20 23 24 26, these were picked by @amoeba, see comments below). Looking at the estimation results obtained re-running FMCD on this amputated data-set, 14 of the 16 observations not classified as outliers by the FMCD fit of the original data-set (and not subsequently removed from the data-set) still are classified as such. The Spearman $\rho$ of the good data went from $0.07$ (p=0.77) on the original data to $0.21$ (p=0.43) on the data set with the four observations selected by @amoeba removed (so the conclusions of the analysis remain the same).
@whuber suggested plotting a scatterplot of the rank of the data to better show the effect of the outliers. The observations flagged as outliers by FMCD are shown as red dots.
@Thomas Speidel. Spearman's $\rho$ has good robustness property, so long as the rate of contamination of the sample is smaller than 5%. For a more in depth (including formal) study of these issues, see [ 3]:
- P. J. Rousseeuw and K. van Driessen (1999) A fast algorithm for the minimum covariance determinant estimator. Technometrics 41, 212–223.
- Ricardo A. Maronna, Douglas R. Martin, Victor J. Yohai (2006).
Robust Statistics: Theory and Methods (Wiley Series in Probability and Statistics).
- Christophe Croux & Catherine Dehon, 2010. "Influence functions of the Spearman and Kendall correlation measures," Statistical Methods and Applications, vol. 19(4), pp 497--515.
