Two logistic regression or one Softmax regression

Question

The following question is from Geron's Hans-On Machine Learning book.

Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you use two Logistic regression or one Softmax regression classifier?

The answer the books gives is that

Need to use two Logistic Regression classifiers since these are not exclusive classes (i.e. all four combinations are possible).

However, the part I don't understand is that we can make them into following four exclusive classes:

1. Daytime and Outdoor
2. Nighttime and Outdoor
3. Daytime and Indoor
4. Nighttime and Indoor This way we still cannot use Softmax regression classifier?

Thanks!

Answer 1

The answer is not clear because 2*2=2+2. I’ll explain. Suppose that you had third dimension winter/summer. In this case for soft max you’d need 8 classes, while with logistics it’s only 6 states in total, I.e. $2\times 3$vs. $2^3$. That’s what he means by non exclusive.

The problem is inherently non exclusive, and if you try yo model it as if it was exclusive you’ll be wasting a lot of resources when the number of dimensions is large. Imagine it’s 20 different binary features. Soft max would have to deal with 1 million states!

Answer 2

If your case was to classify the image as either outdoor, indoor, daytime, nighttime; you'd use softmax. They're now exclusive because any two cannot occur at the same time. But in the original question, a photo can be both daytime and indoor, for example. You should use one logistic regression for each binary classification problem.

If you make four exclusive classes like you listed, then you can certainly use softmax, i.e. last layer of your neural net will contain four neurons responsible for each newly defined class.

Answer 3

I'll answer your question by highlighting a problem which could occur quite often if you use a softmax classifier.

In this case, the output of softmax will be a vector of 4 probabilities in decreasing order. The interpretation of the softmax output is that for a well performing model, the topmost probabilities are the ones that are most likely to be true.

But if you mix together two different phenomenon which are not mutually exclusive, then this interpretation will not be correct.

For example, you could get an output like: Indoor=0.9, Outdoor=0.65, Night=0.35, Day=0.20. If you want to circumvent this problem, an extra logic would need to be applied to this output to re-interpret the two different requirements - Day/Night vs. Indoor/Outdoor.

In case training two classifiers is extremely costly in terms of time/effort then such an approach may still be justified.