I really want to learn about Bayesian techniques, so I have been trying to teach myself a bit. However, I am having a hard time seeing when using Bayesian techniques ever confer an advantage over Frequentist methods. For example: I've seen in the literature a bit about how some use informative priors whereas others use non-informative prior. But if you're using a non-informative prior (which seems really common?) and you find that the posterior distribution is, say, a beta distribution...couldn't you have just fit a beta distribution in the beginning and called it good? I don't see how constructing a prior distribution that tells you nothing...can, well, really tell you anything?
It does turn out that some methods I have been using in R use a mixture of Bayesian and Frequentist methods (the authors acknowledge this is somewhat inconsistent) and I cannot even discern what parts are Bayesian. Aside from distribution fitting, I can't even figure out HOW you would use Bayesian methods. Is there "Bayesian regression"? What would that look like? All I can imagine is guessing at the underlying distribution over and over again while the Frequentist thinks about the data some, eyeballs it, sees a Poisson distribution and runs a GLM. (This isn't a criticism...I really just don't understand!)
So..maybe some elementary examples would help? And if you know of some practical references for real beginners like myself, that would be really helpful too!