# Choosing statistical software based on the size of dataset

The statistical software that I am familiar with, like Stata, is no longer capable for processing 3GB dataset with more than 1 million records, or bigger. Now I am working with JMP pro and a little bit of R.

My slow personal laptop may be a factor too.

Could someone give me a summary of choosing proper statistical software based on the size of dataset?

• I think "G-level" refers to Gigabytes. If you're working with GB sized data, Stata, R, Python (Statsmodels), SAS will all work fine. I believe SAS performs best because the developers allow for some on-disk performance of the data vs simply in-memory like R, Stata and Python use. – Jon Nov 7 '16 at 17:11
• I think the question depends also on the hardware you're using. I have a Dell T7500 with about 60 GB of ram which means I can easily work on files that span a 0-40 GB in size and perform a good amount of work on the data set. However, on my personal laptop, I have about 12GB of RAM, which means that I'm limited to working on files that are approx. 0-10 GB in size. This is just from my own experience, regardless of software. – Jon Nov 7 '16 at 17:14
• 3GB is not big data. Python/R will work fine on data of that size. as john said, ram may make a difference. Another major factor will be what you want to do with that data - some sorts of data manipulation and statistics will have a larger time cost sooner than other types of manipulation/stats. So I don't think there is a clear delineation. Just try to work locally if possible. If it becomes too slow, then 1. ask if you can first filter results down when first querying from your database,m and 2. then maybe distribute processing (spark / hadoop). – captain_ahab Nov 7 '16 at 17:33
• Echoing @Jon: If your Stata can't handle a million observations well, it's because you don't have enough memory. In principle Stata 13 can handle 2 billion observations, and Stata 14 many more, but you must have memory to match for good performance. – Nick Cox Nov 7 '16 at 17:51
• The fact that this question is getting a lot of attention in the form of comments-as-answers suggests it should be considered on topic. The various forms of the answers, though, suggest it might be more suited for CW because no one answer can hope to be comprehensive and uniquely correct: the whole collection of answers will be needed. – whuber Nov 7 '16 at 18:00

I've got to nitpick at the term "big data". The definition varies wildly across people (and industry), but if your data set is less than 40 GB in size, I'd be apprehensive about calling it "big data". I'm Apache Spark dev certified, and the work load that requires tools like Hadoop/Spark are usually distributed data sets in the range of 100GB+. Usually big data already comes with big tools; you wouldn't be installing some piece of software to handle "big data" on your standalone machine.

Now let's focus on large files that span 1-60 GB (from personal experience). I'll call this medium sized data. Data this size will most likely come (extracted) from some SQL database, and will already follow some structure. Data this size will be more common and relevant than the exaggerated talk of "big data".

As mentioned in the comments, I've a Dell Precision T7500 that allows me to comfortably work on data sets as large as 40GB. But let's say your data (.csv, .txt, .json, etc) is only 8GB. Software like R, SAS, Python, Stata will all work fine if your hardware has at least 10GB of memory. Each has performance issues, but if you have an ETL (extract, transform, load) workload that is repetitive with expanding data sets, SAS may work best. By industry standards, SAS is well known to work well with large data sets.

See page 8:

https://support.sas.com/resources/papers/Benchmark_R_Mahout_SAS.pdf

https://support.sas.com/resources/papers/Benchmark-LASR-IMSTAT.pdf

I work regularly with R and Python, and each has it's benefits and costs. R and Python work with data in-memory, meaning that if the file is larger than your RAM, then you will have to find another solution (e.g. SQL database).

R

Recently, Microsoft purchased Revolution Analytics, which gives MS a strong hold in the opensource data science realm.

https://www.microsoft.com/en-us/cloud-platform/r-server

After purchasing Revolution Analytics, MS decided to embed R in their SQL Server 2016 edition. This gives users working with large data sets (but not exactly "big data") some good tools. You can either download SQL Server 2016 with R, or Microsoft Open R which comes with some Intel parallel matrix stuff that OpenBLAS basically already had going but whatever. Parallelizing your matrix computations will be useful if, say, you needed to compute multiple linear regression models or 1 big linear model.

Here is a presentation from the useR! 2016 conference of how R integrates with SQL for working medium sized data

https://channel9.msdn.com/Events/useR-international-R-User-conference/useR2016/Exploring-the-R--SQL-boundary

In working with large data sets, R is slow to read large files. Recent development of readr and data.table allow for faster reading of large files.

https://www.r-bloggers.com/importing-data-into-r-part-two/

Once you've read in the data, you'll most likely need to compute some descriptive statistics and data manipulation/cleaning. With that in mind, I suggest working with dplyr for increased performance.

Python

With the help of the pandas library, Python is very helpful in reading and manipulating data. I've not officially recorded data read times, but Python seems to read as fast as readr (R).

• Only rarely will 10GB be enough to work with an 8GB csv file. You may be able to fread it in but most normal data manipulation will require extra memory (possibly as much again). And of course your OS will need memory too. – Hugh Jun 19 '17 at 14:22