2
$\begingroup$

I am trying to run netlogit on a network of about 60,000 nodes, and I would like to know if the SNA package's functions are designed to support such large operations. I know, for instance, that with RSiena we are not supposed to go further than a few hundred nodes for the algorithms to converge in a reasonable time. But is this the case with SNA's regression models as well?

My initial experience is negative: When I run the commands on my network, the R process expands in memory until it fills both the live memory and the virtual memory and then it crashes (even with one repetition). Is this a matter throwing more resource at R, and is there a good way to calculate how much? Or are the current MRQAP algorithms simply not adapted to such networks? Any resources that can help determine the requirements?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

netlogit() and rmperm() (which is used for netlogit's QAP tests) represent networks with matrices. Representing relatively large networks with matrices will use considerable memory. Not related to memory usage, but the underlying code for netlogit() and rmperm() use for loops within R, which can be slow.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I gave it even 50GBs of RAM and it still crashed. See Carter's response below. $\endgroup$
    – retrography
    Commented May 8, 2015 at 12:32
3
$\begingroup$

Response from Carter Butts:

At present, those functions make heavy use of adjacency representations for the underlying data, and hence do not scale to graphs with more than ~40,000 or so nodes (give or take) - getting around this limit would require a very different implementation. (Generating sufficiently high quality random permutation vectors might also become a problem as N becomes large, since this is a known challenge for pseudorandom number generators. However, I am not aware of anyone who has closely examined the issue in this context, so the impact on real-world performance is hard to assess.) A sparse-data MRQAP implementation for very large graphs could certainly be written, but to make it practical one would need to use an ergm-like term system to calculate covariate values on the fly (since you don't want to store all N^2 values) and whatnot. Doable, but I don't know if anyone has done it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.