4
$\begingroup$

I am trying to calculate the ROC AUC for a dataset where I can't fit predictions and labels in memory (10s/100s billions of samples). Is there a way to calculate the AUC in a distributed way or at least to approximate it?

I know that to calculate the AUC you need to analyse all the possible threshold to calculate the FPR and TPR rates and calculate the area under that curve.

Is there a way to do in a distributed fashion? The only thing I can think of is to calculate the AUC for different batches and then average the different AUCs.

Improve this question
$\endgroup$
9
  • $\begingroup$ How long is that vector you can't fit into memory? $\endgroup$
    – Firebug
    Commented Feb 2, 2018 at 15:28
  • $\begingroup$ I can fit the vector but I don't have all the samples in the same machine. For example: I have 1k samples in one machine and 1k samples in another machine. Is calculating 2 distinct AUCs and then averaging them correct? $\endgroup$
    – Andrea Bergonzo
    Commented Feb 2, 2018 at 15:30
  • $\begingroup$ It's not. Why can't you simply merge the scores into a single vector? $\endgroup$
    – Firebug
    Commented Feb 2, 2018 at 15:36
  • $\begingroup$ Each prediction is a double of 8B and each label is just a boolean of 1b. So if I have 1 billion sample I'll need 1GB only for the scores, right? I was wondering if there is a technique that doesn't rely on having a big memory available. $\endgroup$
    – Andrea Bergonzo
    Commented Feb 2, 2018 at 15:42
  • $\begingroup$ You most probably don't need all 1 billion scores to get reasonable precision on the AUC estimate. $\endgroup$
    – Firebug
    Commented Feb 2, 2018 at 15:45

1 Answer 1

Reset to default
2
$\begingroup$

  • The comments note that sampling will give very precise ROC AUC estimates, even if you only use a fraction of your billions of observations. Working out how tight the confidence set is for your specific problem should be simple enough: load as many randomly-sampled prediction/label pairs as you can into memory and compute the confidence set. If the confidence set is too large for your purposes, you'll need more RAM or to use an out-of-memory method. Luckily, there are simple ways to do either task.

  • You can do the ROC AUC computation without loading all of the data into memory. Suppose that you have a flat file with two columns: predicted value and label. Sort the file by the predicted label in descending order (such as by using the sort command-line utility). Use your preferred method of iterating over a file (command-line utilities, Bash scripting, Python, whatever) to implement this algorithm: https://stats.stackexchange.com/a/146259/22311

  • AWS will let you rent a machine with 384 GB of RAM for less than \$5/hr. In comments, you write that loading 1 billion predicted values into RAM consume take 1GB of memory, so you'll be able to load about 384 billion samples into memory for less than \$5/hr. Using the X1 series is more expensive, but scales to nearly 2000 GB of RAM for less than \$14/hr. (All prices as of this writing.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.