0
$\begingroup$

I'm using the logistic regression of Mahout (version 0.9) but when I check the created model on the same data set it was trained for, I do not see a high value for AUC. I would expect it to be very high since it is the same data set.

My data set is a CSV file with about 7 million lines and has 18 attributes, some numerical and some categorical.

This is how I create the model for logistic regression (I ignore some of the attributes):

$ mahout trainlogistic --input train.csv \
--output ./model \
--categories 2 \
--predictors attribute1 ... attribute15 \
--types w w w n n w w w w w w w n n n \
--target is_delayed \
--rate 100 \
--passes 2 \
--features 500000

And then when I check the AUC value using the model on the same data set:

$ mahout runlogistic --input train.csv --model ./model --auc --confusion
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop, using /usr/lib/hadoop/bin/hadoop and HADOOP_CONF_DIR=/etc/hadoop/conf
MAHOUT-JOB: /usr/lib/mahout/mahout-examples-0.9-cdh5.3.0-job.jar
AUC = 0.48
confusion: [[1703477.0, 761921.0], [3034369.0, 1137161.0]]
entropy: [[NaN, NaN], [-16.5, -17.4]]
15/01/18 06:50:50 INFO driver.MahoutDriver: Program took 98213 ms (Minutes: 1.6368833333333332)

I'm really confused why I only get AUC = 0.48, instead of a value of 1.00 or something very close since it is the same data set.

Do I miss something?

I tried with only a few attributes but still very low AUC, around 0.47, that means the model is almost guessing randomly.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Haven't checked your code but note that there's no general reason to suppose the area under a receiver operating characteristic curve for a given model to be high, even calculated on the training set - why should all models be good? AUC values below 0.5 are possible, though unusual when the model's of even the slightest value, as it's log likelihood that's maximized in fitting rather than AUC. $\endgroup$
    – Scortchi ♦
    Commented Jan 19, 2015 at 9:48

1 Answer 1

Reset to default
2
$\begingroup$

Initially I considered that this question does not belong to this site. However, my notes below are machine-learning related:

  • If --passes 2 means that you want only 2 iterations to run for the iterative logistic algorithm, I expect that only in very rare cases (pathological) this should be enough. I think this is the first thing you have to change.
  • If --rate 100 means the learning rate factor, then again it might be an inappropriate value. I notice that the default value is $0.001$. Consider that having a smaller than optimal value for learning rate might cause very slow convergence; however a learning rate much bigger than optimal might cause the algorithm to not converge at all. Try much smaller values.
Improve this answer
$\endgroup$
2
  • $\begingroup$ According to slideshare.net/tanuvir/logistic-regression-using-mahout and the book "Mahout in Action", "--passes Specifies the number of times the input data should be reexamined during training. Small input files may need to be examined dozens of times. Very large input files probably don’t even need to be completely examined." I assumed 7 million lines with 18 attributes was a very large file, this is why I did "--passes 2". But of course I might be totally wrong. I'll try with more passes. $\endgroup$
    – Emre Sevinç
    Commented Jan 19, 2015 at 10:09
  • $\begingroup$ According to the resources above, "--rate Sets the initial learning rate. This can be large if you have lots of data or use lots of passes because it’s decreased progressively as data is examined." and in the example it is set to 50 (because in the example there are lots of passes). In my case I set it to 100 because I assumed my data was large enough (see above). Again, my assumption might have been totally wrong. $\endgroup$
    – Emre Sevinç
    Commented Jan 19, 2015 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.