softmax equation from Goodfellow's Deep Learning book

Question

I'm having trouble understanding the notation in equation 6.31 Goodfellow/Bengio/Courville's Deep Learning book available here.

The equation is

$$ \text{softmax}(\pmb{z}(\pmb{x},\pmb{\theta}))_i = \frac{\sum_{j=1}^{m}\pmb{1}_{y^{(j)}=i,\pmb{x}^{(j)}=\pmb{x}}}{\sum_{j=1}^{m}\pmb{1}_{x^{(j)}=\pmb{x}}} $$ $m$ is the number of examples in the training set.

They say "Overall, unregularized maximum likelihood will drive the model to learn parameters that drive the softmax to predict the fraction of counts of each outcome observed in the training set"

In the notation section, under functions,they define $\pmb{1}_{condition}$ is 1 if the condition is true and 0 otherwise. So, let us consider three training vectors, $\pmb{x}^{(1)}$,$\pmb{x}^{(2)}$,$\pmb{x}^{(3)}$,corresponding to three classes $y^{(1)}=1,y^{(2)}=2,y^{(3)}=3$. Now we evaluate the ratio for $\pmb{x}=\pmb{x}^{(1)}$ which I think also forces $i=1$

$$ \begin{align} \text{softmax}(\pmb{z}(\pmb{x}^{(1)},\pmb{\theta}))_1 &= \frac{\sum_{j=1}^{m}\pmb{1}_{y^{(j)}=i,\pmb{x}^{(j)}=\pmb{x}}}{\sum_{j=1}^{m}\pmb{1}_{x^{(j)}=\pmb{x}}}\\ &=\frac{\pmb{1}_{y^{(1)}=1,\pmb{x}^{(1)}=\pmb{x}^{(1)}} + \pmb{1}_{y^{(2)}=1,\pmb{x}^{(2)}=\pmb{x}^{(1)}} + \pmb{1}_{y^{(3)}=1,\pmb{x}^{(3)}=\pmb{x}^{(1)}} } {\pmb{1}_{\pmb{x}^{(1)}=\pmb{x}^{(1)}} + \pmb{1}_{\pmb{x}^{(2)}=\pmb{x}^{(1)}} + \pmb{1}_{\pmb{x}^{(3)}=\pmb{x}^{(1)}}}\\ &=\frac{1+0+0}{1+0+0}=1 \end{align} $$

That does not seem correct to me. From their statement I expect the answer to be $1/3$.

Can someone see what I'm doing wrong?

Answer 1

It's normal to have $1/1=1$ if $\pmb{x}^{(i)}\neq \pmb{x}^{(j)}$ for $i\neq j$. If there was fourth sample, which is equal to $\pmb{x}^{(1)}$, i.e. $\pmb{x}^{(4)}=\pmb{x}^{(1)}$, having the same feature values, but it has a different class, say $y^{(4)}=3, $then the softmax equation would approach $1/2$ for the class $1$, $0$ for class $2$, and $1/2$ for class $3$, because it's maximum likelihood.

If you relax the definition a bit, consider the neighborhood around a given sample $\pmb{x}$. The ML estimate is like KNN, $P(y=c|x)$ is calculated by the ratio of samples belonging to class $c$ in that neighborhood to the all samples in that neighborhood. This is represented by Equation 6.31.