I run several simulation studies with R using a MacBook Pro, which has 8 GB of RAM. Unfortunately, some of the simulations cannot be done due to limited memory. My question is how much RAM is needed for large simulations? (For example, each data set has 10000 subjects and needs to generate 1000 datasets). Is 64 GB of RAM big enough for large R simulations?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
Why do you need all 1000 data sets in memory at once? If I were exhausting memory on a lengthy simulation, I'd draw a single simulation, do what I wanted and then throw the simulated data away before moving on to the next simulation
If you write the simulation as a reproducible script then you can always regenerate the exact analysis. If you might need the simulated data again, draw a single simulation, write it out to disk (e.g.
What you describe doesn't seem overly large but it will depend, inter alia, on what data types you have in the data sets and what modelling functions you are using.
The only way anyone can tell if 64GB will be sufficient is you - profile a single simulation and ascertain the memory usage, which should scale linearly over the number of simulations you want to hold in RAM. However, given the size of the data you are generating/using, 8GB should be sufficient if you don't hold all the simulated data sets in memory at once.
Your question is fairly complex, because the answer is: as many as a simulation needs. It depends entirely on the size of your data set, and how you are coding things. I have, for example, simulations that can be run on computers with megabytes of available RAM - and one which crushed a cluster node with 96 GB of memory.
R keeps all its data in memory, which means as you start having huge data sets - or large collections of data sets - you're going to top out your RAM. This system has the benefit of being extremely fast, but limited by RAM. If you're running out of memory resources, recode your project to save these data sets to a file, clear them from R -using rm() - and then open them back up when you need them. This will slow down your code somewhat, but is way cheaper than buying new RAM, especially as a Macbook Pro is going to top out at 16 GB anyway.
The RAM shouldn't be an issue since you usually have unlimited virtual memory. The error "cannot allocate vector of size" is most probably due to the limited address space of your system. A 32-bit system can usually not address more than 4 Gb of memory and I've found that 64-bit seems to work better for R. I don't use Mac but you can check if you're running a 64-bit system in the about menu and then install the R version where you actively choose the 64-bit version.
You could also try just try telling R to allocate more memory:
Here's by the way part of the help on memory.size/limit: