2
$\begingroup$

Assume that output $y$ depends on input $x$ and some noise $\epsilon \sim N(0,\sigma^2)$. $$y = f(x) + \epsilon$$

Suppose we want to model relationship mentioned above using linear neural network:

$$ \hat{y} = w * x + b$$

where $w$ is weight matrix and $b$ is a bias term of neural network.

We can calculate weights using classical analytical solution for OLS:

$$w = (X^TX)^{-1}X^TY$$

Question: how do we calculate (or maybe represent) bias term $b$ in neural network without using gradient descent?

$\endgroup$
2
$\begingroup$

In OLS, $X_{m\times n}$ matrix is the form $\begin{bmatrix}\mathbf x_1 & \dots & \mathbf x_{n-1} & \mathbf 1\end{bmatrix}$ where each element represents an $m\times 1$ vector. First $n-1$ are the feature vectors and the last one is all-1 vector that is to be multiplied with the bias. So, the formula for OLS solves for the bias already using the model:

$$y = X\begin{bmatrix}\mathbf w\\ b\end{bmatrix}+\epsilon$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.