2
$\begingroup$

I was revisiting neural network basics from this post. The perceptron follows below equation:

$$\begin{align} y & = 1 & \text{if } \sum_{i=1}^n w_i\times x_i \geq \theta \\ & = 0 & \text{if } \sum_{i=1}^n w_i\times x_i \lt \theta \\ \end{align} $$

Absorbing threshold value $\theta$ to left hand side, we get:

$$\begin{align} y & = 1 & \text{if } \sum_{i=1}^n w_i\times x_i - \theta \geq 0 \\ & = 0 & \text{if } \sum_{i=1}^n w_i\times x_i - \theta \lt 0 \\ \end{align} $$

Further including $\theta$ in weights and inputs, i.e. $x_0=1$ and $w_0=-\theta$ and starting sum from $i=0$ instead of $i=1$, we get

$$\begin{align} y & = 1 & \text{if } \sum_{\color{red}{i=0}}^n w_i\times x_i \geq 0 \\ & = 0 & \text{if } \sum_{\color{red}{i=0}}^n w_i\times x_i \lt 0 \\ \end{align} $$

I knew this convention of including bias as a first value in weight vector, but it is first time I am finding "threshold value $\theta$" getting moved in as "bias weight $w_0$". Is this what always / normally happens? (that is, does all online articles and book mean the same: including threshold as bias. I never knew bias had any connection with threshold value and was always felt that it is independently learnt during training)

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, each bias parameter can always be interpreted as a threshold for activation.

Another way to think about this is by simply considering how biases affect the activation function. If we have some (activation) function $\phi : \mathbb{R} \to \mathbb{R}$, then $\phi(x - b)$ will be the graph of $\phi(x)$ shifted by an amount $b$ to the right (or left if $b$ is negative). I tried to make a quick desmos demo to visualise this.

The key point is that every activation function indirectly models some sort of "decision boundary" that typically lies at zero. E.g. if the logistic sigmoid function output outputs a value less than one-half, the prediction would be the negative label and if it is greater, we interpret this to be the positive label. However, also for activation functions that are only used in hidden layers (e.g. ReLU), it is often clear that functions behave differently on negative and positive inputs.

TL;DR: By shifting activation functions to the left or the right with the bias parameters, we effectively allow moving around this decision boundary to where we want/need it to be.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.