I'm a mathematician. The following simple example illustrates what I don't understand about finding the weights and bias for a neuron.

Suppose we wish to distinguish the positive numbers from the negatives using a single neuron.

Let's suppose our training set is $x = -1, 1$ with values $y = 0, 1$.
(negative numbers ~ 0 and positive numbers ~ 1)

Let $g(x) = wx + b$.
Let $f(x)$ be the sigmoid function $\dfrac{1}{1 + e^{-x}}$

The neuron's output is $h(x) = f(g(x))$.

If $w > 0$ and $b = 0$ then

$h(x) < 1/2$ for $x < 0$ and

$h(x) > 1/2$ for $x > 0$.

As $w$ increases,

$h(-1) \rightarrow 0$ and $h(1) \rightarrow 1$

So, there is no optimal $w$. As $w$ gets bigger, it fits our training data better, and in fact does what we want (distinguish the negatives from the positives) better and better.

This makes me think that perhaps we would should normalize $w$, say by replacing $(w, b)$ with $(w,b)/||w||$. Here $|| w ||$ is the Euclidean norm on the vector $w$.

It would seem, that in some general sense, that the optimal $w,b$ are defined only up to multiplication by a positive constant. This is similar to the linear equation for a hyperplane being defined only up to multiplication by a non zero constant.

I am sure I am missing something and that this is an obvious issue addressed on day one of neural net class. However, I can't see what I am missing.

Any suggestions are appreciated! Thanks.


2 Answers 2


The optimal $b$ and $w$ are usually chosen to be minimizers (if they exist) of some objective function. in the case of logistic regression (which is your neuron model plus a designated objective function), the objective function to be minimized is the cross-entropy empirical risk function: $$ \label{1} \tag{1} R(w, b \mid \mathcal{D}) = \frac{1}{|\mathcal{D}|} \sum_{(x, y) \in \mathcal{D}} \left(-y \log\left(\frac{1}{1 + e^{-(wx + b)}}\right) - (1 - y) \log\left(\frac{e^{-(wx + b)}}{1 + e^{-(wx + b)}}\right)\right), $$ where $\mathcal{D}$ is your training dataset. This formula is derived from the the Bernoulli log-likelihood function if we make the assumption that the examples in your dataset are i.i.d., and the conditional distribution of your target $y$ given the feature $x$ is $$ y \mid x \sim \operatorname{Bernoulli}\left(\frac{1}{1 + e^{-(wx + b)}}\right). $$ With your posited training set $\mathcal{D} = \{(-1, 0), (1, 1)\}$, the empirical risk objective function becomes $$ \begin{aligned} R(w, b \mid \mathcal{D}) &= -\frac{1}{2} \log\left(\frac{1}{1 + e^{-w-b}}\right) - \frac{1}{2}\log\left(\frac{e^{w - b}}{1 + e^{w - b}}\right) \\ &= \frac{1}{2} \left(\log\left(1 + e^{-w-b}\right) + \log\left(1 + e^{w - b}\right) - w + b\right), \end{aligned} $$ so to find the "optimal" $w$ and $b$ you simply have to find the minimizers of this function.

Unfortunately, it can be shown that this particular empirical risk function does not have a global minimum (or a local one, since \eqref{1} is a convex function), so you are right to say that in this setup there are no optimal $w$ and $b$. You can make the empirical risk $R(w, b \mid \mathcal{D})$ arbitrarily close to $0$ (e.g., by fixing $b$ and making $w$ arbitrarily large).

In practice it is common to augment the empirical risk function by a regularization term. This is a data-independent function $\Omega$ (i.e., it depends only on $w$ and $b$, not on $\mathcal{D}$). Instead of minimizing the empirical risk $R(w, b \mid \mathcal{D})$ directly, one minimizes the sum $$ R(w, b \mid \mathcal{D}) + \Omega(w, b). $$ The role of $\Omega$ is to "discourage model complexity" by penalizing $w$ and $b$ if their absolute values get too large. Common forms for $\Omega$ are the $L^2$ and $L^1$ regularization functions: $$ \begin{aligned} \Omega(w, b) &= \lambda (w^2 + b^2), & \Omega(w, b) &= \lambda (|w| + |b|), \end{aligned} $$ respectively, where $\lambda > 0$ is a hyperparameter that must be chosen prior to estimating $w$ and $b$ to determine how much to penalize large values of $w$ and $b$.

The following pictures demonstrate what happens to the objective function surface when going from no regularization to $L^2$ regularization with increasing values of $\lambda$. In the pictures, blacker regions indicate where the objective function is the smallest (i.e., where near-optimal $w$ and $b$ live).

enter image description here

enter image description here

enter image description here

enter image description here


Normalization using some norm of w is not only done, it's easily available in a lot of ML libraries. Euclidean norm (L2), absolute norm (L1), and a weighted combination of the two are common flavors.

On its own, there is no guarantee - you regularize your way, and/or by hacking the objective function in some other way that penalizes unreasonably scaled weights. Sigmoids (and Softmax, as their generalization) basically have normalization built into the function already.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.