I have been trying to get a sense of the different problems in frequentist settings where MCMC is used. I am familiar that MCMC (or Monte Carlo) is used in fitting GLMMs and in maybe Monte Carlo EM algorithms. Are there more frequentist problems where MCMC is used?
As indicated in the many comments, Markov Chain Monte Carlo is a special case of the Monte Carlo method, which is designed to approximate quantities related with a distribution via pseudo-random number simulation. As such, it has no connection with a particular statistical paradigm and the earliest instances of the method, as in Metropolis et al. (1953), were unrelated with statistics, Bayesian or frequentist. If anything these methods are naturally "frequentist" (an ill-defined category anyway) in that they rely on the stabilisation of the frequencies or averages towards the expectation as the number of simulations increases, aka the Law of Large Numbers.
It is therefore possible within non-Bayesian complex problems to use MCMC methods to replace intractable integrals. Check for instance
- the optimisation of likelihoods with no closed form expressions, as in latent variable and random effect models. The EM algorithm may fail to work because of an intractable "E" step, in which case the expectation need be replaced by a Monte Carlo or a Markov Chain Monte Carlo approximation. With a possible evaluation of the error. Or it may fail to work because of an intractable "M" step, in which case the maximisation can sometimes be replaced by a Markovian maximisation procedure as in simulated annealing. Or using Gibbs steps.
- simulated inference methods in econometrics, as the simulated method of moments, indirect inference, empirical likelihood.
- approximations of likelihoods with intractable normalising constants such as Ising, Potts, and other Markov random fields models, using for instance exchange algorithms.
- frequentist goodness-of-fit tests, which may require computations of coverage probabilities, $p$_values, powers, for sufficient or insufficient statistics with no closed-form density, or conditional on ancillary statistics. Take the example of testing for independence in (large) contingency tables (or deriving the maximum likelihood estimator).
- again in econometrics, Laplace type estimators, "which include means and quantiles of quasi-posterior distributions defined as transformations of general nonlikelihood-based statistical criterion functions, such as those in GMM, nonlinear IV, empirical likelihood, and minimum distance methods" (Chernozhukov and Hong, 2003), rely on MCMC algorithms.