For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for HackersProbabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.
From its description:
An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.
It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code, and is freely available at http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/#contents(in IPython Notebook).