I've read different places that talk about bias terms in neural nets like this
Importance of the bias node in neural networks
But I still have trouble understanding what it is used for and how it affects the neural network.
In a neural network, where would you place it? There can be hundreds of neurons, how do you know which one to link it to? And how many bias terms do you need? How do you know you need one?
Can anyone explain?
Suppose I have a neuron $y = f(w^Tx + b)$ where $x$ is some input, $w$ denotes a set of weights and $b$ represents this bias term you are asking about and of course $f$ is our activation function. I am going to make one more assumption, which seems standard these days, and I'm going to let my activation function be a rectifying linear unit (ReLU), i.e. $f(x) = \max(0, x)$. So $y = \max(0, w^Tx+b)$.
Let's answer the first part of your question in this specific context. Notice that the ReLU will only allow signal to propagate forward if it is greater than 0. So our neuron only "activates" (has a non-zero output value) when $w^Tx + b > 0$ which is equivalent to $w^T x > -b$. So the bias term for a neuron will act as an activation threshold in our setup (ReLU nonlinearities). Since we adaptively learn these bias terms via backpropagation, we may interpret this as we are allowing our neurons to learn when to activate and when not to activate.
To answer your other question about where to place bias terms, observe that the bias is built into the formula.
If you do not want to include bias terms, you would simply design your network to be of the form $y = f(w^T x)$ for example.
I suspect what you are getting at is: how do I know when to include a bias term or not? As you mentioned, neural networks, if you really view them on a neuron per neuron basis, the different types of parameterizations (for example, why should every neuron have the same activation function?) of neural networks becomes overwhelming.
Luckily there is a Universal Approximation Theorem which basically states that a single hidden layer neural network can approximate any arbitrary continuous function, provided we have enough hidden units. Therefore, even though we are using seemingly uniform units (by uniform I do not mean that each of the units all have the same weights, just that they are all from the same "class") all of the form $y = f(w^T x+ b)$, if we have enough of these hidden units, we can approximate all sorts of complex, nonlinear functions to any desired accuracy. Note that the Universal Approximation Theorem that I referenced is for sigmoidal activation functions.
Lastly, note that just because such a neural network is guaranteed to exist, that doesn't mean it is easy to find via training/optimization. See Yoshua Bengio and Yann Dauphin's article Big Neural Networks Waste Capacity, which touches on this issue and explores the diminishing returns issue when constructing bigger or deeper neural networks.
It seems impractical to try to parameterize your neural network on a neuron per neuron basis. Modern nets have millions of parameters (easily). Therefore, the deep learning community has basically decided to trust in the Universal Approximation Theorem and not try to tackle the overwhelming problem of neuron by neuron design. Instead, much of the deep learning innovations have been in more broader network design (i.e. convolutional layers, recurrent layers, how to stack these, etc.), and optimization issues (SGD+Momentum, weight initialization, Dropout regularization, data pre-processing, data augmentation, and lots and lots of other issues) as the Universal Approximation Theorem is an existence result and not a constructive result: it doesn't tell us how to construct or find this nice approximator. It doesn't even tell us how many units we will need if we want to approximate a continuous function to some fixed degree of accuracy.
