The Bayesian approach proposes using our previous expereince or knoweldge, if available. For example, If I ask you what is the probability of having a head in a coin flip, you will probably say 50%. However, if you actually flip the coin 3 times, you might get 1/3, 2/3 or even 3/3 heads.
From a frequentists point of view (opposed to Bayesians), the probabilities would be (1/3 ~= 33%, 66%, 100%). The Baysians, on the other hand, say that we already previously know that the probability of having a head in a coin flip is 50% (our prior knoweldge). Let's use it! Now, the probabilities can be corrected by this prior factor so that, the probability of having 1/3 heads = (1/3)*(1/2) = 16% (and not 33% as we got before).
From a first glance, the Baysian approach seems very attractive as it allows us to make use of our prior knowedlge. However, if we are not able to quatify our prior knoweldge with good estimates, then, we might deviate from the actual description of our surrounding environment.
A small comment for the Gaussain distribution choice, is that it simplifies learning the parameters of the model. In particualr, it can be shown that if the distribution of the prior knowedlge is Gaussian, then the posterior distribution is also a Gaussain (conjugate prior). Needless to say, this might not be the best justification to use a Gaussian model as the assumptions of the Gaussian distribution might not hold for a particular dataset, yet, it performs well in many cases.