What would be a good way to work with a large data set in Excel? I have a large data set which is in .dbf format right now and what I would like to do is be able to manipulate it easily in Excel and do something like subtotal and calculate stdev and ratios.
Details of the data set;
This data set contains shopper information. It has 1.2 million rows and 20 columns where the rows are each a unique shopper and the columns hold their shopping data (what they bought).
I am using Office 2007 programs, I know Excel the best but was wondering what alternatives I could use to accomplish my goals (subtotal, calculate stdev, and ratio's).
 A: If you feel you may start more of such very large Excel type projects in the future, then you should consider installing and spending 10 hours learning the basics of R (free), which will let you do what you mention in your question, in a much more efficient manner than Excel.
R for Beginners PDF
You can ask questions about R on StackOverflow and here.
A: Although I would always recommend to use R, you could nevertheless achieve what you want with python.
There is at least a package for reading dbf files.
Furthermore, scipy offers a great range of functions for statistical analysis. For example the library ScientifyPython probably contains the functions you need.
The best idea is to check scipy.org. There you will find what you want.
(But learning R is a great idea!!)
A: Excel 2010 and 2013 have a free microsoft addin called  power pivot which allows you to work with millions of rows. Its a columnar database that is designed for creating pivot tables, subtotals etc and has standard deviation etc predefined. you might also look at other microsoft addins power query (data input), power view (visualisation), and power map (mapping)
