# Distributed AUC calculation (or approximation)

I am trying to calculate the ROC AUC for a dataset where I can't fit predictions and labels in memory (10s/100s billions of samples). Is there a way to calculate the AUC in a distributed way or at least to approximate it?

I know that to calculate the AUC you need to analyse all the possible threshold to calculate the FPR and TPR rates and calculate the area under that curve.

Is there a way to do in a distributed fashion? The only thing I can think of is to calculate the AUC for different batches and then average the different AUCs.

• How long is that vector you can't fit into memory? – Firebug Feb 2 '18 at 15:28
• I can fit the vector but I don't have all the samples in the same machine. For example: I have 1k samples in one machine and 1k samples in another machine. Is calculating 2 distinct AUCs and then averaging them correct? – Andrea Bergonzo Feb 2 '18 at 15:30
• It's not. Why can't you simply merge the scores into a single vector? – Firebug Feb 2 '18 at 15:36
• Each prediction is a double of 8B and each label is just a boolean of 1b. So if I have 1 billion sample I'll need 1GB only for the scores, right? I was wondering if there is a technique that doesn't rely on having a big memory available. – Andrea Bergonzo Feb 2 '18 at 15:42
• You most probably don't need all 1 billion scores to get reasonable precision on the AUC estimate. – Firebug Feb 2 '18 at 15:45

• You can do the ROC AUC computation without loading all of the data into memory. Suppose that you have a flat file with two columns: predicted value and label. Sort the file by the predicted label in descending order (such as by using the sort command-line utility). Use your preferred method of iterating over a file (command-line utilities, Bash scripting, Python, whatever) to implement this algorithm: https://stats.stackexchange.com/a/146259/22311
• AWS will let you rent a machine with 384 GB of RAM for less than \$5/hr. In comments, you write that loading 1 billion predicted values into RAM consume take 1GB of memory, so you'll be able to load about 384 billion samples into memory for less than \$5/hr. Using the X1 series is more expensive, but scales to nearly 2000 GB of RAM for less than \\$14/hr. (All prices as of this writing.)