Representing bias term in simple linear neural network(linear regression) using analztical solution

Question

Assume that output $y$ depends on input $x$ and some noise $\epsilon \sim N(0,\sigma^2)$. $$y = f(x) + \epsilon$$

Suppose we want to model relationship mentioned above using linear neural network:

$$ \hat{y} = w * x + b$$

where $w$ is weight matrix and $b$ is a bias term of neural network.

We can calculate weights using classical analytical solution for OLS:

$$w = (X^TX)^{-1}X^TY$$

Question: how do we calculate (or maybe represent) bias term $b$ in neural network without using gradient descent?

Answer 1

In OLS, $X_{m\times n}$ matrix is the form $\begin{bmatrix}\mathbf x_1 & \dots & \mathbf x_{n-1} & \mathbf 1\end{bmatrix}$ where each element represents an $m\times 1$ vector. First $n-1$ are the feature vectors and the last one is all-1 vector that is to be multiplied with the bias. So, the formula for OLS solves for the bias already using the model:

$$y = X\begin{bmatrix}\mathbf w\\ b\end{bmatrix}+\epsilon$$