Distributed AUC calculation (or approximation) I am trying to calculate the ROC AUC for a dataset where I can't fit predictions and labels in memory (10s/100s billions of samples). Is there a way to calculate the AUC in a distributed way or at least to approximate it?
I know that to calculate the AUC you need to analyse all the possible threshold to calculate the FPR and TPR rates and calculate the area under that curve.
Is there a way to do in a distributed fashion? The only thing I can think of is to calculate the AUC for different batches and then average the different AUCs.
 A: *

*The comments note that sampling will give very precise ROC AUC estimates, even if you only use a fraction of your billions of observations. Working out how tight the confidence set is for your specific problem should be simple enough: load as many randomly-sampled prediction/label pairs as you can into memory and compute the confidence set. If the confidence set is too large for your purposes, you'll need more RAM or to use an out-of-memory method. Luckily, there are simple ways to do either task.

*You can do the ROC AUC computation without loading all of the data into memory. Suppose that you have a flat file with two columns: predicted value and label. Sort the file by the predicted label in descending order (such as by using the sort command-line utility). Use your preferred method of iterating over a file (command-line utilities, Bash scripting, Python, whatever) to implement this algorithm: https://stats.stackexchange.com/a/146259/22311 

*AWS will let you rent a machine with 384 GB of RAM for less than \$5/hr. In comments, you write that loading 1 billion predicted values into RAM consume take 1GB of memory, so you'll be able to load about 384 billion samples into memory for less than \$5/hr. Using the X1 series is more expensive, but scales to nearly 2000 GB of RAM for less than \$14/hr. (All prices as of this writing.)
