6
$\begingroup$

Please help interpret results of logistic regression produced by weka.classifiers.functions.Logistic from the WEKA library.

I use numeric data from WEKA examples:

@relation weather

@attribute outlook {sunny, overcast, rainy}
@attribute temperature real
@attribute humidity real
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

@data
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
rainy,70,96,FALSE,yes
rainy,68,80,FALSE,yes
rainy,65,70,TRUE,no
overcast,64,65,TRUE,yes
sunny,72,95,FALSE,no
sunny,69,70,FALSE,yes
rainy,75,80,FALSE,yes
sunny,75,70,TRUE,yes
overcast,72,90,TRUE,yes
overcast,81,75,FALSE,yes
rainy,71,91,TRUE,no

To create the logistic regression model I use the following command:

java -cp WEKA_INS/weka.jar weka.classifiers.functions.Logistic -t WEKA_INS/data/weather.numeric.arff -T WEKA_INS/data/weather.numeric.arff -d ./weather.numeric.model.arff

Here are what the three arguments mean:

-t <name of training file> : Sets training file.
-T <name of test file> : Sets test file. 
-d <name of output file> : Sets model output file.

Running the above command produced the following output:

Logistic Regression with ridge parameter of 1.0E-8
Coefficients...
              Class
Variable                    yes
===============================
outlook=sunny           -6.4257
outlook=overcast        13.5922
outlook=rainy           -5.6562
temperature             -0.0776
humidity                -0.1556
windy                    3.7317
Intercept                22.234

Odds Ratios...
              Class
Variable                    yes
===============================
outlook=sunny            0.0016
outlook=overcast    799848.4264
outlook=rainy            0.0035
temperature              0.9254
humidity                 0.8559
windy                   41.7508


Time taken to build model: 0.05 seconds
Time taken to test model on training data: 0 seconds

=== Error on training data ===
Correctly Classified Instances          11               78.5714 %
Incorrectly Classified Instances         3               21.4286 %
Kappa statistic                          0.5532
Mean absolute error                      0.2066
Root mean squared error                  0.3273
Relative absolute error                 44.4963 %
Root relative squared error             68.2597 %
Total Number of Instances               14     

=== Confusion Matrix ===
 a b   <-- classified as
 7 2 | a = yes
 1 4 | b = no

Questions:

  1. First section of the report:

    // Coefficients...
    
                  Class
    Variable                    yes
    ===============================
    outlook=sunny           -6.4257
    outlook=overcast        13.5922
    outlook=rainy           -5.6562
    temperature             -0.0776
    humidity                -0.1556
    windy                    3.7317
    Intercept                22.234
    
    • Do I understand right that Coefficients are in fact weights that are applied to each attribute before adding them together to produce the value of class attribute play equal to yes?
  2. Second section of the report:

    // Odds Ratios...
    
                  Class
    Variable                    yes
    ===============================
    outlook=sunny            0.0016
    outlook=overcast    799848.4264
    outlook=rainy            0.0035
    temperature              0.9254
    humidity                 0.8559
    windy                   41.7508
    
    • What is the meaning of "Odds Ratios"?

    • Do they all also relate to class attribute play equal to yes?

    • Why is the value outlook=overcast so much bigger than the value of outlook=sunny?

  3. Confusion matrix

    === Confusion Matrix ===
     a b   <-- classified as
     7 2 | a = yes
     1 4 | b = no
    
    • What is the meaning of "Confusion Matrix"?

Thanks a lot for your help!

$\endgroup$
9
$\begingroup$

Let me explain what odds mean in general.

Odds are the ratio between the probability of success over the probability of failure, that is, $\displaystyle \frac{p_{i}}{1-p_{i}}$. Let's say $p_{i}$ for a given event is 0.6, then the odds for that event is $0.6/0.4=1.5$.

1- As you said, since the logistic regression outputs probabilities based on the following equation:

$$\text{logit}(p_{i}) = \log{\frac{p_{i}}{1-p_{i}}} = \beta_{0} + \beta_{1}x_{1} + ... + \beta_{k}x_{k}$$

the coefficients refer to each $\beta_{i}$.

2- Odds ratios are simply the exponential of the weights you found before. For example, the first coefficient you have is outlook=sunny: -6.4257. If you calculate $\exp(-6.4257)$ you get 0.0016 that is the corresponding value in the odds ratio table.

The relation between the coefficient for outlook=sunny and its odds ratio is, in this case, the logarithm of the odds of outlook=sunny over the odds of outlook=¬sunny: $$\displaystyle \log{\frac{Odds(outlook=sunny)}{Odds(outlook=¬sunny)}}$$

For instance, the odds of outlook=sunny is the probability of a sunny day in which you can play over the probability of having a sunny day in which you can't play. Similarly, you can calculate the odds for outlook=¬sunny. The log of this ratio is the value of the coefficient attached to the variable outlook=sunny in the logistic regression. However, in this particular example, since you have more than one variable as predictors, it's necessary to fix the value of the other variables. Now you can see why outlook=overcast has such a value. The odds for outlook=overcast are extremely favorable to the yes outcome, producing a high positive value.

A simpler example of this can be found here.

3.- The confusion matrix is very simple. In the first row, for example, it tells you the number of instances classified in your training data as yes that you classified as yes (that is, 7) and the number that are classified as yes that you classified as no(2). The second row is equivalent for instances classified as no.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Any chance to get a p-value from the output? $\endgroup$ – qed Dec 7 '14 at 20:22
  • $\begingroup$ Do you mean the p-value of the coefficients for the weather dataset? $\endgroup$ – Robert Smith Dec 7 '14 at 20:27
  • $\begingroup$ Yeah, precisely. :-) $\endgroup$ – qed Dec 7 '14 at 20:28
  • 1
    $\begingroup$ The OP did the logistic regression in Weka and if I remember correctly, I didn't rerun his example. It shouldn't take you a lot of time to obtain the output in R or Python. I'm not sure why the output in Weka didn't include p-values. $\endgroup$ – Robert Smith Dec 7 '14 at 20:56
  • $\begingroup$ What lib/libs do you recommend for doing this in python? I am looking for something with high-performance. $\endgroup$ – qed Dec 7 '14 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.