# A proof of uniqueness of weights and bias for a neural net

I'm a mathematician. The following simple example illustrates what I don't understand about finding the weights and bias for a neuron.

Suppose we wish to distinguish the positive numbers from the negatives using a single neuron.

Let's suppose our training set is $$x = -1, 1$$ with values $$y = 0, 1$$.
(negative numbers ~ 0 and positive numbers ~ 1)

Let $$g(x) = wx + b$$.
Let $$f(x)$$ be the sigmoid function $$\dfrac{1}{1 + e^{-x}}$$

The neuron's output is $$h(x) = f(g(x))$$.

If $$w > 0$$ and $$b = 0$$ then

$$h(x) < 1/2$$ for $$x < 0$$ and

$$h(x) > 1/2$$ for $$x > 0$$.

As $$w$$ increases,

$$h(-1) \rightarrow 0$$ and $$h(1) \rightarrow 1$$

So, there is no optimal $$w$$. As $$w$$ gets bigger, it fits our training data better, and in fact does what we want (distinguish the negatives from the positives) better and better.

This makes me think that perhaps we would should normalize $$w$$, say by replacing $$(w, b)$$ with $$(w,b)/||w||$$. Here $$|| w ||$$ is the Euclidean norm on the vector $$w$$.

It would seem, that in some general sense, that the optimal $$w,b$$ are defined only up to multiplication by a positive constant. This is similar to the linear equation for a hyperplane being defined only up to multiplication by a non zero constant.

I am sure I am missing something and that this is an obvious issue addressed on day one of neural net class. However, I can't see what I am missing.

Any suggestions are appreciated! Thanks.

The optimal $$b$$ and $$w$$ are usually chosen to be minimizers (if they exist) of some objective function. in the case of logistic regression (which is your neuron model plus a designated objective function), the objective function to be minimized is the cross-entropy empirical risk function: $$\label{1} \tag{1} R(w, b \mid \mathcal{D}) = \frac{1}{|\mathcal{D}|} \sum_{(x, y) \in \mathcal{D}} \left(-y \log\left(\frac{1}{1 + e^{-(wx + b)}}\right) - (1 - y) \log\left(\frac{e^{-(wx + b)}}{1 + e^{-(wx + b)}}\right)\right),$$ where $$\mathcal{D}$$ is your training dataset. This formula is derived from the the Bernoulli log-likelihood function if we make the assumption that the examples in your dataset are i.i.d., and the conditional distribution of your target $$y$$ given the feature $$x$$ is $$y \mid x \sim \operatorname{Bernoulli}\left(\frac{1}{1 + e^{-(wx + b)}}\right).$$ With your posited training set $$\mathcal{D} = \{(-1, 0), (1, 1)\}$$, the empirical risk objective function becomes \begin{aligned} R(w, b \mid \mathcal{D}) &= -\frac{1}{2} \log\left(\frac{1}{1 + e^{-w-b}}\right) - \frac{1}{2}\log\left(\frac{e^{w - b}}{1 + e^{w - b}}\right) \\ &= \frac{1}{2} \left(\log\left(1 + e^{-w-b}\right) + \log\left(1 + e^{w - b}\right) - w + b\right), \end{aligned} so to find the "optimal" $$w$$ and $$b$$ you simply have to find the minimizers of this function.
Unfortunately, it can be shown that this particular empirical risk function does not have a global minimum (or a local one, since \eqref{1} is a convex function), so you are right to say that in this setup there are no optimal $$w$$ and $$b$$. You can make the empirical risk $$R(w, b \mid \mathcal{D})$$ arbitrarily close to $$0$$ (e.g., by fixing $$b$$ and making $$w$$ arbitrarily large).
In practice it is common to augment the empirical risk function by a regularization term. This is a data-independent function $$\Omega$$ (i.e., it depends only on $$w$$ and $$b$$, not on $$\mathcal{D}$$). Instead of minimizing the empirical risk $$R(w, b \mid \mathcal{D})$$ directly, one minimizes the sum $$R(w, b \mid \mathcal{D}) + \Omega(w, b).$$ The role of $$\Omega$$ is to "discourage model complexity" by penalizing $$w$$ and $$b$$ if their absolute values get too large. Common forms for $$\Omega$$ are the $$L^2$$ and $$L^1$$ regularization functions: \begin{aligned} \Omega(w, b) &= \lambda (w^2 + b^2), & \Omega(w, b) &= \lambda (|w| + |b|), \end{aligned} respectively, where $$\lambda > 0$$ is a hyperparameter that must be chosen prior to estimating $$w$$ and $$b$$ to determine how much to penalize large values of $$w$$ and $$b$$.
The following pictures demonstrate what happens to the objective function surface when going from no regularization to $$L^2$$ regularization with increasing values of $$\lambda$$. In the pictures, blacker regions indicate where the objective function is the smallest (i.e., where near-optimal $$w$$ and $$b$$ live).