Problem: Prove that: \begin{align} \Delta E(in) &= -\frac{1}{N} \sum_{n=1}^N \frac{y_n x_n}{1 + e^{(y_n w^t x_n)}} \\[10pt] &=\frac{1}{N} * \sum_{n=1}^N - y_n x_n \theta (-y_n w^T x_n) \end{align}

Problem Description: I'm trying to prove this equality but I'm finding that my answer is not making any sense to me. Does anyone have a clue about this equality?

My doubt is about $\theta$, if theta is the log regression, then how the $...+e^{-...}$ is supposed to become positive? $...+e^{+...}$

Also, $e$ from the logistic regression, has only -x, and in the formula there are a lot of variables!

On The Equality

The denominator is being 'powered' by $y_n w^T x_n$. However, on the logistic regression, $e ^{-x}$. So how is logistic regression supposed to have more variables than X? Unless I consider that X can be anything. Which is obvious to me, but I'm not sure. And about the sign "-/+"

Edit: The questions goes further, inferring that this equality would make "misclassified" examples contribute more to the gradient. However, logistic regression just calculates p(x|y), so how is that supposed to change anything? Also the range is just from $0\to 1$, so negative values wouldn't change either, since they aren't supposed to be inserted anyway.

So what about that? Unless I could add a variable to penalize misclassified values, I fail to see how that equality changes anything.

Conclusion This was my first question here, and I did the tour and learning the rules and applying them! Sorry for anything!


Wikipedia - Logistic Regression

  • 1
    $\begingroup$ Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Please make these changes as just posting your homework & hoping someone will do it for you is grounds for closing. $\endgroup$ – Silverfish Jul 1 '16 at 1:46
  • $\begingroup$ It would also help to include a little more of the context for this question - particularly to define your variables. $\endgroup$ – Silverfish Jul 1 '16 at 1:46
  • $\begingroup$ Will do @Silverfish ! I just need a bit more time! $\endgroup$ – KenobiShan Jul 1 '16 at 21:00
  • $\begingroup$ How can i approve the edits? Do i need to aprove anything? Thanks for editing! I really appreciate! $\endgroup$ – KenobiShan Jul 1 '16 at 21:01

I suppose $θ$ is the logistic function $θ(x) = \frac{1}{1+e^{-x}}$. Now, let's rewrite your initial formula like this (notice I also moved the minus into the summation): $$ ΔE(in) = \frac{1}{N}*\sum\limits_{i=1}^N(-y_nx_n * \frac{1}{1+e^{-(-y_nw^Tx_n)}}) $$ It is clear that $\frac{1}{1+e^{-(-y_nw^Tx_n)}} = θ(-y_nw^Tx_n)$. Substituting this into the first equation we get the needed equality: $$ ΔE(in) = \frac{1}{N}*\sum\limits_{i=1}^N(-y_nx_n * θ(-y_nw^Tx_n)) $$ Please ask if you need further help.

| cite | improve this answer | |
  • 1
    $\begingroup$ I was stuck with the "-" sign ! , i was going through the wrong way! Now i see everything makes sense and its so simple! Thanks so much! Does this change changes how the gradient will be calculated? $\endgroup$ – KenobiShan Jul 1 '16 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.