3
$\begingroup$

In this question on stackoverflow, I asked about how it is possible to find the individual significance of each correlation coefficient of each node. I answered the question myself later stating that we can use the normalized values of the product of the variables to identify the individual significance. However, I am not sure now whether this is correct or not. The summation that leads to the final correlation coefficient depends on the single values of the product of the normalized x and y. Some of the single values are higher than others which leads to the individual significance.

The reason why I am asking about this because the data are indexed based on the nodes as shown in the table. So I am assuming that the data of each node can have its own effect. For example, all the nodes are contributing to the positive correlation, but node 242 was the reason the correlation coefficient was not 1 but 0.9. So, knowing this, I can isolate and investigate node 242.

nodes   closeness   degree      actual_relays
238     0.622695    0.394077    0.0799
242     0.654735    0.472665    0.0791
247     0.653274    0.476082    0.0673
250     0.648928    0.458998    0.0689
254     0.705788    0.583144    0.1056
259     0.660647    0.486333    0.1125

My background in statistics is mediocre so please correct me as well. Is the method described viable mathematically? If not, then what is the way to go about it? Any references are appreciated.

$\endgroup$
  • $\begingroup$ How is the "individual significance" of each node defined? $\endgroup$ – David Jul 2 '19 at 15:05
  • $\begingroup$ The summation that leads to the final correlation coefficient depends on the single values of the product of the normalized x and y. Some of the single values are higher than others which leads to the individual significance. I will try to addd more details to the questions and please correct me if I am wrong. $\endgroup$ – Ahmed Al-haddad Jul 2 '19 at 21:13
4
$\begingroup$

It is natural to compute the "influence" or "sensitivity" of individual data points on a statistical procedure by leaving them out of the dataset, redoing the procedure, and examining how its result has changed. (One hesitates to use the word "significance" for this assessment because that word has a special meaning in statistics.)

This approach is general and usually readily interpretable; the main issue usually concerns how best to measure the change. Arguably, a change in correlation is better assessed in terms of its Fisher z transform instead of a raw difference or ratio.

Here is an example of a bivariate dataset of 128 points. The "influence" is simply the difference in Z-transforms of the correlations. Points are scaled by the absolute value of the influence and colored according to whether including a point has increased the correlation (red) or decreased it (blue). The black line is the least-squares fit to all points, shown for reference.

Figure

The R code that produced the graphics includes a function cor.sensitivity implementing this approach.

#
# Given parallel vectors, return an array of the influences of each 
# component on their correlation.
#
cor.sensitivity <- function(x, y, use.xform=TRUE) {
  if (isTRUE(use.xform)) {
    xform <- function(r) log((1+r)/(1-r))/2 # The Fisher Z transform
  } else {
    xform <- identity
  }
  n <- length(x)
  j <- which(!is.na(x) & !is.na(y))
  x <- x[j]
  y <- y[j]
  delta <- sapply(seq_along(j), function(i) xform(cor(x[-i], y[-i])))
  rho <- rep(xform(cor(x, y)), n)
  rho[-j] <- NA_real_
  if(length(j) > 0) rho[j] <- rho[j] - delta
  rho
}
#
# Generate sample data.
#
n <- 128
rho <- -1/2
set.seed(17)
x <- c(rnorm(n-1), 5)
y <- rho * x + sqrt(1-rho^2) * rnorm(n)
#
# Compute the influence/sensitivity values.
#
delta <- cor.sensitivity(x, y)
#
# Plot the results.
#
library(ggplot2)

X <- data.frame(x=x, y=y, influence=delta, influence.a=abs(delta))
names(X)[4] <- "|influence|"
ggplot(X, aes(x, y, size=`|influence|`, fill=influence)) + 
  geom_point(shape=21) + 
  geom_smooth(method=lm, se=FALSE, col="Black") + 
  scale_fill_gradient2(trans="reverse")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.