How to read large dataset in R [closed]

Question

trying to read a large dataset in R the console displayed the follwing errors:

data<-read.csv("UserDailyStats.csv", sep=",", header=T, na.strings="-", stringsAsFactors=FALSE)
> data = data[complete.cases(data),]
> dataset<-data.frame(user_id=as.character(data[,1]),event_date= as.character(data[,2]),day_of_week=as.factor(data[,3]),distinct_events_a_count=as.numeric(as.character(data[,4])),total_events_a_count=as.numeric(as.character(data[,5])),events_a_duration=as.numeric(as.character(data[,6])),distinct_events_b_count=as.numeric(as.character(data[,7])),total_events_b=as.numeric(as.character(data[,8])),events_b_duration= as.numeric(as.character(data[,9])))
Error: cannot allocate vector of size 94.3 Mb
In addition: Warning messages:
1: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :
  NAs introduced by coercion
2: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :
  NAs introduced by coercion
3: In class(value) <- "data.frame" :
  Reached total allocation of 3583Mb: see help(memory.size)
4: In class(value) <- "data.frame" :
  Reached total allocation of 3583Mb: see help(memory.size)

Does anyone know how to read large datasets? The size of UserDailyStats.csv is approximately 2GB.

Answer 1

Two basic things:

1. That complaint addresses all of the memory in the R session, not just the one object that you're loading. And unless you're using something like ff, everything in your session is in memory.
2. One Windows, you need to specify how much memory can be used by R. Have a look at help(memory.limit). Even though you're using 64-bit, it won't default to use all your available memory.

Otherwise, you can consider using bigmemory to handle larger datasets if it's still a problem. Some relevant sources:

• The High Performance Computing view on CRAN.
• Dirk Eddelbuettel's tutorial on the subject.
• Ryan Rosario's presentation on the subject.

As a final suggestion, you can try calling gc() to free up memory before running your command, although in principle R will do this automatically as it needs to.

Answer 2

What platform are you running R on? How much physical and virtual memory does the machine have?

Also, you might find the following relevant: http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html

Answer 3

Since you're on 64-bit Windows, make sure that you have installed and are running the 64-bit version of R for Windows. Then, follow the instructions on Gary King's page:

• How do I increase the memory for R?

Answer 4

I,m totally agree with Dirk answer. One suggestion. I have found very useful the use of programming languages such as AWK or others when assessing large databases. So, I was able to filter the data I wanted to include in my analysis, reducing the final size of dataset.

Moreover, in your code you are duplicating the same data set twice (data and dataset). If you want to define your variables as factor, numeric, etc, you could use of colClasses option in the read.table function.

Answer 5

You can always use the filehash package which dumps the large dataset in the disk rather than in the system's memory. You need to be prepared to deal with the tradeoff though, due to slow read HDD times.