3
$\begingroup$

I would like to demonstrate the "over-parameterization" of the softmax function and its relation to the sigmoid function with a practical example.

With toy data, it's easy to show that:

logits = np.array([.123, .456])

softmax(logits) == np.array([ sigmoid(logits[0] - logits[1]), 1 - sigmoid(logits[0] - logits[1]) ])

With real data, I'm constructing both a vanilla logistic regression model and vanilla k=2 softmax regression model, each without a bias term. All weights are initialized to .0001. I'm running 1 step of gradient descent, using a batch size of 1.

Should the two functions make identical predictions? What relationship, if any, should we observe between the weights and/or logits of the respective functions?

Empirically, I'm not seeing any relationship between the two.

$\endgroup$
1

1 Answer 1

3
$\begingroup$

Let $Y \in \{0,1\}$ be the outcome variable and let $X$ denote a single predictor. Without the bias term, the softmax functions can be written as:

$P(Y=0|X=x) = \dfrac{e^{x\beta_0}}{e^{x \beta_0}+e^{x\beta_1}} = \dfrac{1}{1+e^{x(\beta_1-\beta_0)}}$

$P(Y=1|X=x) = \dfrac{e^{x\beta_1}}{e^{x \beta_0}+e^{x\beta_1}} = \dfrac{e^{x(\beta_1-\beta_0)}}{1+e^{x(\beta_1-\beta_0)}}$

Hence, the probabilities only depend on the difference $\beta_1 - \beta_0$. Choosing $\beta_0=8$ and $\beta_1=10$, for example, will give you the same value of the logistic loss function as $\beta_0=108$ and $\beta_1=110$. Both parameters are therefore not individually identifiable.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.