How to do statistical analysis on very large CSV files (over 200GB)? Does anyone know how to do statistical analysis on very (very) large CSV files? I'm currently building a dataset for my prof, and once joined together it'll be a CSV file with size >200GB, 92 million rows and 75 columns. I know R base functions and readr can't import such a large file (tried and failed on a 90GB CSV). Is there any way I can do stats stuff on such a large dataset? Ideally a solution using R would be good but I'm open to other options as well.
 A: I would start by looking into the data.table package. It allows for fast reads and can do many operations very fast, (benchmarks).
The problem with data.table, and base R, is that you typically load all the dataset into memory. Given that your file is 200GB, you will need at least that much memory. Quite frankly, unless the task you are solving really needs all the data in memory, then this could be a waste of resources. I would at least try to learn about the other options.
Other options, which are less memory exhaustive, is to use a database, (some SQL database, e.g. sqlite3). You can use the dbplyr package to query the database with dplyr commands. This will consume much less memory, and you can create indices to make certain queries faster. The database will still take quite a lot of disc space, and some queries will be slower, since it could require a lot of random access reads.
Another option is to split the original file into smaller files, e.g. if there is a timestamp, split them by year, so one file for 2001, one for 2002, 2003, etc. Then you will to write a bit of extra code to query the data, but you will at least be able to load and access the data in R.
You can also save the smaller batches with base::save, which saves the file into an R binary file, which can be faster to load in many cases.
The solution really depends on the problem that you want to solve. Also consider whether you really need all these 75 columns, maybe you can get away with fewer and that saves a lot of space.
If it really is the case, that you need all the data, e.g. if you are doing clustering or something like that. Then I would still start with a smaller subset of the data to develop and iterate on. If you start on all the data, then the development cycle will be very long, in case you want to change, tweak or test something different.
Hope this helps!
