Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I run several simulation studies with R using a MacBook Pro, which has 8 GB of RAM. Unfortunately, some of the simulations cannot be done due to limited memory. My question is how much RAM is needed for large simulations? (For example, each data set has 10000 subjects and needs to generate 1000 datasets). Is 64 GB of RAM big enough for large R simulations?

share|improve this question
13  
How long is a piece of string? –  Gavin Simpson Jul 7 '11 at 18:01
9  
Slightly more helpful... Why do you need all 1000 data sets in memory at once? Draw a simulation do what you want and throw the simulated data away. If you write the simulation as a reproducible script then you can always regenerate the exact analysis. If you might need the simulated data, draw a single simulation, write it out to disk, discard the one in RAM and draw another, etc. What you describe doesn't seem large... Perhaps more information would help people comment. Also this is OT for this forum. StackOverflow would be better but you need to improve the Q first. –  Gavin Simpson Jul 7 '11 at 18:03
1  
I think Gavin actually answered your question, indeed write each of the generated data set as .csv or whatever format you want and rm() it at once. –  Dmitrij Celov Jul 8 '11 at 0:03
    
Hi thanks to both of you first. When I run a simulation which has 1000 dataset and each data set has 30000 subjects. Soon, the R console shows: Error: cannot allocate vector of size 686.6 Mb. –  Tu.2 Jul 8 '11 at 4:19
1  
You might get better suggestions if you actually post some code. –  curious_cat Mar 1 '13 at 14:35

3 Answers 3

Why do you need all 1000 data sets in memory at once? If I were exhausting memory on a lengthy simulation, I'd draw a single simulation, do what I wanted and then throw the simulated data away before moving on to the next simulation

If you write the simulation as a reproducible script then you can always regenerate the exact analysis. If you might need the simulated data again, draw a single simulation, write it out to disk (e.g. save() or write.csv()), discard the one in RAM and draw another, etc.

What you describe doesn't seem overly large but it will depend, inter alia, on what data types you have in the data sets and what modelling functions you are using.

The only way anyone can tell if 64GB will be sufficient is you - profile a single simulation and ascertain the memory usage, which should scale linearly over the number of simulations you want to hold in RAM. However, given the size of the data you are generating/using, 8GB should be sufficient if you don't hold all the simulated data sets in memory at once.

share|improve this answer
    
+1 for moving comments :) –  Brandon Bertelsen Sep 3 '11 at 19:56

Your question is fairly complex, because the answer is: as many as a simulation needs. It depends entirely on the size of your data set, and how you are coding things. I have, for example, simulations that can be run on computers with megabytes of available RAM - and one which crushed a cluster node with 96 GB of memory.

R keeps all its data in memory, which means as you start having huge data sets - or large collections of data sets - you're going to top out your RAM. This system has the benefit of being extremely fast, but limited by RAM. If you're running out of memory resources, recode your project to save these data sets to a file, clear them from R -using rm() - and then open them back up when you need them. This will slow down your code somewhat, but is way cheaper than buying new RAM, especially as a Macbook Pro is going to top out at 16 GB anyway.

share|improve this answer

The RAM shouldn't be an issue since you usually have unlimited virtual memory. The error "cannot allocate vector of size" is most probably due to the limited address space of your system. A 32-bit system can usually not address more than 4 Gb of memory and I've found that 64-bit seems to work better for R. I don't use Mac but you can check if you're running a 64-bit system in the about menu and then install the R version where you actively choose the 64-bit version.

Update:

You could also try just try telling R to allocate more memory:

# Current memory limit
memory.limit()

# Set limit to max 32-bit
# Limit must be less than: 2^32/2^20
memory.limit(4095)

# Check that it did what you wanted
memory.limit()

Here's by the way part of the help on memory.size/limit:

If 32-bit R is run on most 64-bit versions of Windows the maximum value of obtainable memory is just under 4Gb. For a 64-bit versions of R under 64-bit Windows the limit is currently 8Tb.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.