I have an adjacency matrix and another which represents whether the two nodes share an attribute. Consider it like an homophily test. We want to test if the likelihood to form a connect depends on the fact that the two nodes have an attribute in common. Now, using R and SNA package, I run a correlation and test is significance through a QAP test:

g <- array(dim=c(2,nrow(x),nrow(x)))
g[1,,] <- x
g[2,,] <- y
q.12 <- qaptest(g, gcor, reps = 2000, g1=1, g2=2, diag=FALSE)

The correlation is 0.7479487, and the p-value is 0

QAP Test Results

Estimated p-values:
    p(f(perm) >= f(d)): 0 
    p(f(perm) <= f(d)): 1 

Then I fit a logit to that data

nl <- netlogit(y, x, mode="digraph", diag=FALSE, nullhyp="qap", reps=2000)

but its coefficient is not significant. How is that possible?

Network Logit Model

            Estimate  Exp(b)       Pr(<=b) Pr(>=b) Pr(>=|b|)
(intercept) -2.940634 5.283224e-02 0.000   1.000   0.00     
x1          21.506702 2.188981e+09 0.519   0.481   0.97     

Goodness of Fit Statistics:

Null deviance: 17234.41 on 12432 degrees of freedom
Residual deviance: 4617.118 on 12430 degrees of freedom
Chi-Squared test of fit improvement:
     12617.29 on 2 degrees of freedom, p-value 0 
AIC: 4621.118   BIC: 4635.974 
Pseudo-R^2 Measures:
    (Dn-Dr)/(Dn-Dr+dfn): 0.5036986 
    (Dn-Dr)/Dn: 0.7320989 

I have asked on the SNA mailing list, and prof. Carter Butts courteously replied as follows:

Hi, Raffaele -

The short answer is that no, these situations do not have to give you the same answer. They are different tests of different quantities against different null hypotheses, and as such do not have to agree.

The longer answer requires looking in more detail at what you are doing (and your data). In your first case, you are running a bivariate QAP test of the hypothesis that the observed value of the graph correlation between x and y was drawn from the matrix permutation distribution of the graph correlation.

In your second case, you have a (network) logistic regression of y on x and an intercept, and are testing the z-scores for each of those coefficients against a version of the QAP SPP null hypothesis.

These are different things. In most cases they will give you similar answers, but not always. In your case switching the test statistic from the z-score to the raw coefficient gives different results, which is a clue to where the discrepancy lies. So is the large coefficient on x, and the whopping nominal standard error on that coefficient. If we look at a 2x2 table of x by y values, we see that your data has an entailment: x perfectly implies y. As a result, you're on the face of the convex hull of the sufficient statistics, and the MLE doesn't exist. We can confirm that using ergm:

Observed statistic(s) edgecov.x are at their greatest attainable values. Their coefficients will be fixed at +Inf. Starting maximum pseudolikelihood estimation (MPLE): Evaluating the predictor and response matrix. Maximizing the pseudolikelihood. Finished MPLE. Stopping at the initial estimate. Evaluating log-likelihood at the estimate.

MLE Coefficients: edges edgecov.x -2.941 Inf

The netlogit function doesn't have ergm's tricks for recognizing convex hull problems, so it cheerily tries to maximize the likelihood along the direction of recession until the numerical optimizer runs out of steam. But it leaves clues for us, e.g. giving us a coefficient that might as well be infinite on the logit scale, and giving us a huge nominal se (which is why the z-score test isn't significant, BTW).

So we can now see why netlogit is giving surprising results: there's no MLE for this model. So anything else you get from it is going to be ad hoc.

By contrast we have no such problem with your first analysis, because we're just computing a sample statistic (and its permutation distribution). There's nothing inherently problematic about having an exact entailment: it just happens to be at the extremes of the possible data from the point of view of the network logit family.

BTW, if you fit a network OLS model, you will see that you get back the results you expect. That's also an exponential family, but a different one, and your data is not on the convex hull of the statistics for that. All is hence copacetic.

Hopefully this clears things up. It's a good reminder that details matter - and that when you get surprising results, it's often a good idea to take a close look at your data.

Hope that helps,



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.