So I understand the bias term as essentially akin to the slope in regular multiple linear regression, my question is how do I add one to my data? Is it as simple as adding a column with all 1's to my training data set?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ "slope"? should it be "intercept"? $\endgroup$– Haitao DuCommented Jul 24, 2017 at 2:13
-
$\begingroup$ Yeah sorry my bad that was what I meant! $\endgroup$– no neinCommented Jul 27, 2017 at 15:27
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Yes, you can add a column with 1's to your data and treat it like a regular feature, but this will only add the bias term to the first layer. You would have to create an extra dimension with an 1's for every layer. You can also create variables for the biases and treat them separately from the other weights. Check out this numpy implementation of a neural network from scratch for details.
By the way, where are you implementing your neural network? If you only need to use it and not implement it by yourself, you can use some higher level library such as keras.
-
$\begingroup$ I am just fiddling around with it for fun. I am a little unclear what you mena with creating an extra dimension with 1s for the second layer? $\endgroup$– no neinCommented Jul 24, 2017 at 13:18
-
$\begingroup$ You have to add the bias term to the hidden layer as well. What I meant is that you can do this by adding a new node with activation fixed to 1, and treat it like a regular node. The weight associated with it will be the bias term for this layer. However, I prefer to treat the bias term separately from the other weights, like in the example I sent. $\endgroup$ Commented Jul 29, 2017 at 17:23
$\begingroup$
$\endgroup$
In the linear part of the layer, an extra column of ones (for each layer) serves as a bias term, because it is a linear combination of not just the inputs but also a constant term (not a function of the inputs).
Consider the use of homogeneous coordinates in graphics and computer vision, where adding an extra dimension allows affine transformations instead of simply linear transformations.
