If Bayesian and Frequentist approaches give different answer, why can't we settle the matter by experiment once and for all? As we have seen, Bayesian and Frequentist approaches sometimes give different answers to real world problem. So why can't we perform experiments and determine (with a very high likelyhood) who is right? Or is it the case that sometimes the Frequentist is right, sometimes the Bayesian?
 A: This is a most interesting question in that it does not have an answer. The major issue with statistics is that there is no "truth" in realistic settings and thus that reaching or uncovering this "truth" is impossible. Given a dataset, and an assumed model (that is most likely "wrong"), it is possible to describe the probabilistic properties of a statistical procedure with regard to this model. But depending on the procedure, the model, and the property, one procedure can be better or worse.
An example is provided by classical decision theory: let us assume an observation $x$ that is taken as a realisation of a Normal $\text{N}(\theta,1)$ model and let us settle on evaluating errors by a simple squared difference $(d-\theta)^2$ [all of those are assumptions and choices, there is nothing absolute in those!]. The error cannot be minimised uniformly over all values of $\theta$. If one settles for an average error, one can pick the frequentist risk
$$R(\delta,\theta)=\int_\mathcal{X} (\delta(x)-\theta)^2 \varphi(x-\theta)\text{d}x$$
or the Bayesian risk
$$r(\delta,\pi) = \int_\Theta (\delta(x)-\theta)^2 \pi(\theta|x)\text{d}\theta$$
the later allowing for an optimal solution that depends on the prior $\pi$. While the former does not allow for an optimal solution for all values of $\theta$. One must then consider minimaxity or admissibility to sift though procedures. And even then there may be several minimax or several admissible procedures, most of which will be Bayesian.
You could take a look at the discussion of another question: Concrete-examples-of-a-frequentist-approach-that-is-superior-to-a-Bayesian-one
