Why does sklearn and tensorflow give different ROC AUC scores?

Question

My tensorflow ML algorithm gives me an ROC AUC of 0.81 using the contrib.metrics.streaming_auc() function, whereas using the same logits and labels in sklearn's function gives me a score of 0.58. How can this be?

When I reduce the number of thresholds in the tensorflow function, tensorflow too gives me significantly lower AUC, so can the explanation be in the two libraries different ways of calculating the AUC?

Relevant code snippets, first from building the tensorflow graph:

...
self.logits = tf.matmul(last_layer_outputs, weights) + bias
self.loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
                                     self.logits,
                                     tf.argmax(self.y_placeholder,1)))
...
self.softmaxed_logits = tf.nn.softmax(self.logits)
(_, self.auc_update_op) = tf.contrib.metrics.streaming_auc(
                              predictions=self.softmaxed_logits,
                              labels=self.y_placeholder,
                              curve='ROC')

Then from asking sklearn.metrics and tensorflow for the AUC:

...
soft_logits = model_instance.sess.run(model_instance.softmaxed_logits,
                                      feed_dict=test_feed)
sklearn_auc = metrics.roc_auc_score(y_true=model_instance.T3SS_data.y_test,
                                    y_score=soft_logits)
tf_auc = model_instance.sess.run(model_instance.auc_update_op,
                                 feed_dict=test_feed)

Answer 1

The streaming_auc keeps accumulating the scores of repeated calls to it, so that you can use it, for example, to get the AUC of several batch runs all accumulated. It does not just calculate the current AUC.

In order to use it to get just the current AUC, you can reset the local variables it uses (e.g. running tf.initialize_local_variables()) before running its update operation.

Answer 2

NO ! its because Scikit learn does not calculate AUC by discretizing a curve with variable thresholds like is done in tensorflow! SKlearn uses your values

this has just burnt me.