How to interpret Weka Logistic Regression output?

Question

Please help interpret results of logistic regression produced by weka.classifiers.functions.Logistic from the WEKA library.

I use numeric data from WEKA examples:

@relation weather

@attribute outlook {sunny, overcast, rainy}
@attribute temperature real
@attribute humidity real
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

@data
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
rainy,70,96,FALSE,yes
rainy,68,80,FALSE,yes
rainy,65,70,TRUE,no
overcast,64,65,TRUE,yes
sunny,72,95,FALSE,no
sunny,69,70,FALSE,yes
rainy,75,80,FALSE,yes
sunny,75,70,TRUE,yes
overcast,72,90,TRUE,yes
overcast,81,75,FALSE,yes
rainy,71,91,TRUE,no

To create the logistic regression model I use the following command:

java -cp WEKA_INS/weka.jar weka.classifiers.functions.Logistic -t WEKA_INS/data/weather.numeric.arff -T WEKA_INS/data/weather.numeric.arff -d ./weather.numeric.model.arff

Here are what the three arguments mean:

-t <name of training file> : Sets training file.
-T <name of test file> : Sets test file. 
-d <name of output file> : Sets model output file.

Running the above command produced the following output:

Logistic Regression with ridge parameter of 1.0E-8
Coefficients...
              Class
Variable                    yes
===============================
outlook=sunny           -6.4257
outlook=overcast        13.5922
outlook=rainy           -5.6562
temperature             -0.0776
humidity                -0.1556
windy                    3.7317
Intercept                22.234

Odds Ratios...
              Class
Variable                    yes
===============================
outlook=sunny            0.0016
outlook=overcast    799848.4264
outlook=rainy            0.0035
temperature              0.9254
humidity                 0.8559
windy                   41.7508


Time taken to build model: 0.05 seconds
Time taken to test model on training data: 0 seconds

=== Error on training data ===
Correctly Classified Instances          11               78.5714 %
Incorrectly Classified Instances         3               21.4286 %
Kappa statistic                          0.5532
Mean absolute error                      0.2066
Root mean squared error                  0.3273
Relative absolute error                 44.4963 %
Root relative squared error             68.2597 %
Total Number of Instances               14     

=== Confusion Matrix ===
 a b   <-- classified as
 7 2 | a = yes
 1 4 | b = no

Questions:

1. First section of the report:

   // Coefficients...
   
                 Class
   Variable                    yes
   ===============================
   outlook=sunny           -6.4257
   outlook=overcast        13.5922
   outlook=rainy           -5.6562
   temperature             -0.0776
   humidity                -0.1556
   windy                    3.7317
   Intercept                22.234
   

   • Do I understand right that Coefficients are in fact weights that are applied to each attribute before adding them together to produce the value of class attribute play equal to yes?

2. Second section of the report:

   // Odds Ratios...
   
                 Class
   Variable                    yes
   ===============================
   outlook=sunny            0.0016
   outlook=overcast    799848.4264
   outlook=rainy            0.0035
   temperature              0.9254
   humidity                 0.8559
   windy                   41.7508
   

   • What is the meaning of "Odds Ratios"?

   • Do they all also relate to class attribute play equal to yes?

   • Why is the value outlook=overcast so much bigger than the value of outlook=sunny?

3. Confusion matrix

   === Confusion Matrix ===
    a b   <-- classified as
    7 2 | a = yes
    1 4 | b = no
   

   • What is the meaning of "Confusion Matrix"?

Thanks a lot for your help!

Answer 1

Let me explain what odds mean in general.

Odds are the ratio between the probability of success over the probability of failure, that is, $\displaystyle \frac{p_{i}}{1-p_{i}}$. Let's say $p_{i}$ for a given event is 0.6, then the odds for that event is $0.6/0.4=1.5$.

1- As you said, since the logistic regression outputs probabilities based on the following equation:

$$\text{logit}(p_{i}) = \log{\frac{p_{i}}{1-p_{i}}} = \beta_{0} + \beta_{1}x_{1} + ... + \beta_{k}x_{k}$$

the coefficients refer to each $\beta_{i}$.

2- Odds ratios are simply the exponential of the weights you found before. For example, the first coefficient you have is outlook=sunny: -6.4257. If you calculate $\exp(-6.4257)$ you get 0.0016 that is the corresponding value in the odds ratio table.

The relation between the coefficient for outlook=sunny and its odds ratio is, in this case, the logarithm of the odds of outlook=sunny over the odds of outlook=¬sunny: $$\displaystyle \log{\frac{Odds(outlook=sunny)}{Odds(outlook=¬sunny)}}$$

For instance, the odds of outlook=sunny is the probability of a sunny day in which you can play over the probability of having a sunny day in which you can't play. Similarly, you can calculate the odds for outlook=¬sunny. The log of this ratio is the value of the coefficient attached to the variable outlook=sunny in the logistic regression. However, in this particular example, since you have more than one variable as predictors, it's necessary to fix the value of the other variables. Now you can see why outlook=overcast has such a value. The odds for outlook=overcast are extremely favorable to the yes outcome, producing a high positive value.

A simpler example of this can be found here.

3.- The confusion matrix is very simple. In the first row, for example, it tells you the number of instances classified in your training data as yes that you classified as yes (that is, 7) and the number that are classified as yes that you classified as no(2). The second row is equivalent for instances classified as no.