I've been working on the topic of regularization for neural networks and in the textbook I'm following I found this quote:

"We will further simplify the analysis by making a quadratic approximation to the objective function in the neighborhood of the value of the weights that obtains minimal unregularized training cost. The approximation of J is given by

$$\hat J(\theta) = J(w^*) + 1/2(w − w^∗)^\intercal H(w − w^∗)$$

where H is the Hessian matrix of J with respect to w evaluated at w*."

Now I'm working on quadratic approximations in terms of calculus, but can somebody please explain how do we take quadratic approximation of the function J, if it is defined as usual mean squared error:

$$J(\theta) = \frac{1}{2n}\sum_{i=1}^n (y_n - \hat y_n)^2$$

I mean, how do we get this approximation of J defined earlier? Thank you in advance.


closed as unclear what you're asking by Sycorax, Michael Chernick, kjetil b halvorsen, COOLSerdash, jbowman Jul 2 '18 at 0:17

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ If $J$ is already quadratic, why do you want to approximate it with a quadratic function? The 2nd degree Taylor expansion of a quadratic function is simply the original quadratic function. Am I missing something here? $\endgroup$ – Matthew Gunn Dec 11 '16 at 0:46
  • $\begingroup$ @MatthewGunn it is written in the textbook that quadratic approximation is used to simplify the function... Anyway, I think that I'm getting the idea of this thing except one: do we assume that function is parameterized with vector W, containing w and b terms? I mean, to create the Hessian matrix, we need to have 2 parameters of the function to take derivatives. Tell me if I'm wrong somewhere. $\endgroup$ – olejnik_ Dec 11 '16 at 10:08