# How to read large dataset in R [closed]

trying to read a large dataset in R the console displayed the follwing errors:

data<-read.csv("UserDailyStats.csv", sep=",", header=T, na.strings="-", stringsAsFactors=FALSE)
> data = data[complete.cases(data),]
> dataset<-data.frame(user_id=as.character(data[,1]),event_date= as.character(data[,2]),day_of_week=as.factor(data[,3]),distinct_events_a_count=as.numeric(as.character(data[,4])),total_events_a_count=as.numeric(as.character(data[,5])),events_a_duration=as.numeric(as.character(data[,6])),distinct_events_b_count=as.numeric(as.character(data[,7])),total_events_b=as.numeric(as.character(data[,8])),events_b_duration= as.numeric(as.character(data[,9])))
Error: cannot allocate vector of size 94.3 Mb
1: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :
NAs introduced by coercion
2: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :
NAs introduced by coercion
3: In class(value) <- "data.frame" :
Reached total allocation of 3583Mb: see help(memory.size)
4: In class(value) <- "data.frame" :
Reached total allocation of 3583Mb: see help(memory.size)


Does anyone know how to read large datasets? The size of UserDailyStats.csv is approximately 2GB.

-
Why do you insist on crossposting here and on SO? –  Dirk Eddelbuettel Oct 19 '10 at 15:46
Given that you shamelessly cross-post, allow me to equally shamelessly refer you to my answer to your identical question on StackOverlow –  Dirk Eddelbuettel Oct 19 '10 at 15:48
-1; No need to be acerbic. –  rpierce Nov 22 '10 at 7:04

## closed as off topic by Andy W, gung, whuber♦Jan 9 at 13:51

Questions on Cross Validated are expected to relate to statistics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Two basic things:

1. That complaint addresses all of the memory in the R session, not just the one object that you're loading. And unless you're using something like ff, everything in your session is in memory.
2. One Windows, you need to specify how much memory can be used by R. Have a look at help(memory.limit). Even though you're using 64-bit, it won't default to use all your available memory.

Otherwise, you can consider using bigmemory to handle larger datasets if it's still a problem. Some relevant sources:

As a final suggestion, you can try calling gc() to free up memory before running your command, although in principle R will do this automatically as it needs to.

-

What platform are you running R on? How much physical and virtual memory does the machine have?

Also, you might find the following relevant: http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html

-
I'm running Windows 7 64-bit on Quad CPU 2.67 GHz, 4GB RAM –  niko Oct 19 '10 at 13:07

Since you're on 64-bit Windows, make sure that you have installed and are running the 64-bit version of R for Windows. Then, follow the instructions on Gary King's page:

-
You can always use the filehash package which dumps the large dataset in the disk rather than in the system's memory. You need to be prepared to deal with the tradeoff though, due to slow read HDD times.