Bias is a measure of the model's in-sample fitting ability. Generally, a more flexible model will have a lower bias (ie it fits the data well). The problem with low-bias models is that they can fit the data too well (ie. they start fitting the noise in the data too).
This where the need of adding some discipline to the model arises. So, in some cases like lasso or ridge regression, people regularize the model by adding a penalty to a non-zero parameter estimate. This is the same as adding a prior for the model parameter (for example, ridge regression is equivalent to linear regression with a normal prior, of mean zero and certain variance, for parameters).
A prior is used to incorporate null hypothesis/expert opinion/theoretical insights on the distribution of model parameters. This is often used to ensure that a very flexible model (which has an inclination to fit the data well) doesn't come up with estimates which "don't make sense" ie. are driven by noise in the sample (for example, if my model estimates the probability of "getting a head" to be 1, it's possibly because I have a sample which doesn't represent the population well. So, I make it hard for the model to come up with such an estimate).
In fact, it can be shown, at least in the simple case of the linear regression, that adding a prior is equivalent to adding additional data generated under our null hypothesis (so in your example we are adding a few tosses of a fair coin) to the sample collected from the external world. So, you are right, a prior does bias the model towards a certain outcome. But this is often seen as a disciplining bias.