I spent a lot of time thinking about your problem and how to answer it. I can actually point out problems which either only have Bayesian solutions, or for which the Bayesian solution is better in a radical way, but I am concerned it wouldn't be useful to you because I think you are really having a mental fog with the problem. I think it is the framework you are putting the solutions in. I will give you a trivial example later that you can look at, but I don't think it is the issue involved.
As a background note, almost my entire formal training is either Pearson and Neyman's Frequentist methods or Fisher's Likelihood method. I have some formal Bayesian training, but almost as a side note. Because of this, I recognize in your question the same types of problems I faced when I ran into a problem with no good non-Bayesian solution. My foray into Bayesian thought was an accident. Also because of this, I try really hard to get people to use the tool that actually is best for the problem at hand and not the one they know. This means sometimes I talk people out of Bayesian methods or into them. This same thinking also applies to things such as artificial neural networks. They are a tool and sometimes a good tool and sometimes inferior.
I am going to start with your edit. The quote:
I am asking this question because I learnt about the Bayesian method and I feel there is nothing much more it gives compared with the more efficient MLE, except for an (not so good) estimation of confidence interval.
This is a serious misunderstanding. I am going to work from back to front on this one. Bayesian credible intervals are not Pearson and Neyman confidence intervals. They do not have the same interpretation, implied meanings, or use.
A confidence interval is probably a misnomer. It would better to say "the interval that is a result of a confidence procedure." The confidence is in the procedure and not the result. A 95% confidence interval of (a,b) is implied to cover a parameter, $\theta$, at least 95% of the times the procedure is performed. This is critically important in several ways.
The first and obvious one is that it says nothing about the true location of $\theta,$ except that if you behave as if $\theta$ is in the interval, then you are guaranteed you will not be made a fool of at least 95% of the time, but cannot say that there is a 95% chance that the true value is inside the interval $(a,b).$ Indeed, you can show cases where there is a 100% chance that it cannot be in the interval. Without using a Bayesian method, you can state nothing about your specific sample. There is either a 100% or a 0% chance it is in the interval from Pearson and Neyman's perspective. You do not know which one it is.
If you want a better understanding of confidence intervals and credible intervals I recommend two reads. The first is here on stack exchange, the second in a journal article.
The first is an excellent presentation by Keith Weinstein on the how the two different types of intervals compare to each other under a simple example and why, in some cases, they do not match at all. You can find it at Confidence versus Credible Intervals and Cookies
The second shows cases where the Frequentist result is incredibly inaccurate. You are interpreting confidence intervals as a measure of precision. There is a separate method for calculating precision. They do not measure precision. Let me give you a simple example. Imagine you had the entire population of one school and you split it 50-50 between two researchers in an entirely random manner. Each researcher estimated the sample mean and variance of some parameter of the children.
Without loss of generality assume $\sigma^2_1<\sigma^2_2$. The confidence interval of researcher one will be smaller than researcher two if a textbook standard confidence interval were used. Yet the only difference was the random selection. What about being selected into group one made it a more accurate representation of the mean? It isn't. A smaller interval is not more precise than a wider interval in any intrinsic sense of the idea. The boundaries of a confidence interval are a computation from the random numbers generated by the sample, so the boundaries are random numbers.
Morey, R., Hoekstra, R., Rouder, J., Lee, M., & Wagenmakers, E.-J. (2015). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 1–21
To use an extreme case from the Morey article above, consider a submarine sunk on the bottom of the ocean. It is ten meters long. You get one chance to find the sub's rescue hatch or they die. The submarine produces two bubbles, but they produced anywhere along the length of the submarine with uniform probability and only once. The hatch is in the middle.
If two bubble rise and they are together, then the Bayesian likelihood is 10 meters wide and the 50% interval is the point of the bubbles plus or minus 2.5 meters. The Frequentist interval implied from the uniformly most powerful test has zero width, which implies extreme accuracy precisely when it is most inaccurate. Conversely, when the bubbles are exactly 10 meters apart the Bayesian likelihood is of zero width and there is a 100% chance that the hatch is at the midpoint of the bubbles. However, the Frequentist interval implied by the uniformly most powerful test is at its widest of $\bar{x}\pm\frac{10}{2}.$
Conversely, the 95% credible interval is an interval with a 95% chance of containing the parameter. Unfortunately, as you can see in the cookie example above, it may not contain the true value 95% of the times you repeat an experiment. The percentage could be quite small. Indeed, the cookie example above includes a case where the 70% credible interval will, upon repetition, only contain the true answer 20% of the time if there were no Bayesian updating.
The first part of the quote was the MLE was more accurate. This is not correct. The MLE is the most likely point estimate. To use Stigler's quote (see reference below):
At a superficial level, the idea of maximum likelihood must be prehistoric: early hunters and gatherers may not have used the words ``method of maximum likelihood'' to describe their choice of where and how to hunt and gather, but it is hard to believe they would have been surprised if their method had been described in those terms. It seems a simple, even unassailable idea: Who would rise to argue in favor of a method of minimum likelihood, or even mediocre likelihood?
Stigler, Stephen M. The Epic Story of Maximum Likelihood. Statist. Sci. 22 (2007), no. 4, 598-620. doi:10.1214/07STS249. https://projecteuclid.org/euclid.ss/1207580174
Being the most likely solution is not the same thing as being the most accurate solution. Stein's lemma shows that this is not the case. See:
Stein, C. (1956). "Inadmissibility of the usual estimator for the mean of a multivariate distribution". Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. 1. pp. 197–206.
Indeed, if accuracy is your goal, then as long as you use your best estimate of the prior density for the parameters, all Bayesian estimators are guaranteed to be admissible. Non-Bayesian estimators are only admissible if they either map to the Bayesian estimator in every sampling or converge at the limit. If accuracy is your goal, then Bayesian estimators cannot be stochastically dominated by non-Bayesian estimators. Non-Bayesian estimators can be stochastically dominated by Bayesian estimators.
However, the goal may not be accuracy. It may be to assure an absence of bias. If the goal is not accuracy, but protection from bias, a guaranteed level of protection against false positives and an ability to control the level of false negatives, then you should use Pearson and Neyman's minimum variance unbiased estimator. Generally, the MLE is not unbiased and with a proper prior the Bayesian estimator is nearly guaranteed to be biased.
What does the MLE provide then if not unbiasedness or accuracy, it provides a solution to the epistemological question of whether a null hypothesis is likely to be false by providing a measure of the "weight of the evidence against" the null. That weight is called the p-value. Pearson and Neyman use an acceptance region or a rejection region based on a cutoff value $\alpha$ and so the p-value contains no additional information in that case. In the Likelihoodist methodology, however, you are looking at the strength of the evidence with the p-value.
Now let me give you a simple case that I have provided more fully elsewhere on stack exchange, but I cannot find now. Nonetheless, I will let you solve it computationally and you will find that there may be serious unexpected consequences by your choice of method.
Imagine that the Congress authorized the creation of biased and unbiased US quarters. The bias for heads is either $\{1/3,1/2,2/3\}.$ The probability of being randomly chosen from the minted coins is $\{25\%,50\%,25\%\}$ for each respective choice. You can either construct a solution where you are ignorant of the prior distribution or you know it.
Now as a short thought experiment you were a bookie. You can either choose the MLE or the Bayesian solution. The Bayesian solution is a discrete choice among the three possible parameters, but the MLE is just $\frac{k}{n}$ or the number of heads divided by the number of trials.
Let us imagine you tossed the coin 8 times to gather information on the true value of the parameter. You saw the results as well as the gamblers. Based only on this sample, what odds would you post for each possible outcome for a future set of 8 trials? What if you saw six heads or zero heads?
For zero heads, the MLE is for a double-tailed coin with no variance at all. The MAP estimator is for a $1/3$ bias under a uniform prior. If you use the Bayesian and Frequentist predictive distributions rather than plug in the MLE or MAP estimator you will get a big increase in Bayesian accuracy, but except in the corner cases no increase in the Frequentist accuracy. Indeed, the Frequentist odds are so misstated that you can guarantee that you can place arbitrage bets against the bookie and never lose.
Choose Bayesian methods when accuracy is your primary concern or when gambling is involved. Use MLE when you need to reject a false understanding of the world. Use Pearson and Neyman methods when you need a clean estimate that provides you some guaranteed level of protection from making the wrong decisions while controlling for enough power to be able to say you are making a good decision.
To give you a regression example that I have run. Go to the FDIC database of bank financial statements from before the crisis. Look at the impact of Federal Reserve tightening or loosening on risk-taking behavior as a function of leverage. You will get very different parameter results due to the fact that some banks fail, some are started, and some just have missing information at times. Because of how Bayesian updating works, you will end up with different parameter estimates in the sense that both the values and some of the signs are different, and the accuracy is different.
Although that is an example, if I were just learning, I wouldn't try that problem. Build a toy problem, as above, with the coins. The simplest solution to get large differences is to build a boundary condition in. Most non-Bayesian regression methods presume the entire real number line is a possible value, but if there are known restrictions they will include impossible answers. It can be the case, sometimes, that you can build restrictions into a regression by limiting support, but not always and not without lots of headaches. Bayesian methods just assign a 0% chance in impossible regions.
