Skip to main content
Tweeted twitter.com/StackStats/status/809969057579601920
edited title
Link
Haitao Du
  • 37.3k
  • 25
  • 148
  • 244

Stochastic gradient descent for regularized logistic regression

Source Link
wwl
  • 718
  • 1
  • 7
  • 19

Stochastic gradient descent for logistic regression

At 8:30 of this video Andrew Ng mentions that the cost function for stochastic gradient descent (for a single observation) for logistic regression is

$-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i) + \frac{\lambda}{2} ||w||^2$

My question (a rather technical one) is about the regularization term. If the cost function for all observations is

$\sum_{i=1}^n \{-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i)\} + \frac{\lambda}{2} ||w||^2$

should the cost function for a single observation be

$-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i) + \frac{\lambda}{2n} ||w||^2$

? In other words, the regularization term is divided by $n$; it is "spread out" across all observations. I know its rather technical as $\lambda$ can easily be changed, but I want to make sure I get the concept right.