Let's assume the tossing of an unfair coin is modeled by a random variable X taking the values head and tail. You know that the objective probability of the coin showing up head is either $p=0.4$ or $p=0.6$ and that no other value is possible. Further, you have reason to believe that $p=0.6$ is very likely to be true.
Following the Bayesian methodology, you could express your belief by the (subjective) prior distribution
$$\begin{align} P(p=0.6)&=0.9 \\ P(p=0.4)&=0.1\text{,} \end{align}$$
hence treating $p$ as a random variable. Once you do this, marginalization in combination with Lewis's principle implies that your subjective probability for the coin showing up head should be
$$P(X=\text{head})=0.6 \times 0.9+0.4 \times 0.1=0.58\text{,}$$
which is rather inappropriate even as subjective probability, because after all you know that $p$ can either take the value $0.4$ or $0.6$ and hence your subjective probability should either be $0.4$ or $0.6$.
I would like to know how Bayesians respond to that issue. Has this issue been discussed theoretically, and what is the Baysian position towards it?
@Arya McCarthy: Thank you for your answer. I think you are right that Baysians interpret P(X=head)=0.58 as degree of belief, that heads will show up. My question is why does this makes sense, and according to which criterion? I feel this boils down to the justification of the prior. Imagine a Bayesian also has the same prior distribution for other coins as well and that the objective probability of getting a coin with p=0.6 is actually 0.9 and the objective probability of getting a coin with p=0.4 is actually 0.1. If in an experiment, the Baysian selects a series of coins at random and tosses them, $P(X=\text{heads})=0.58$ would be justified, because in this two stage random experiment (first select a coin and then toss it) the overall relative frequency of head will roughly speaking converge to the expected value, which is $0.58$. Hence, roughly speaking, a long run strategy based on $P(X=\text{head})=0.58$ would actually be successful in the real world and this seems to me as a good criterion for P(X=head)=0.58 to make sense, - actual long run success considering similar situations. So how do you arrive at a prior, that roughly reflects these two objective probabilities? You could collect a sample and calculate frequentist estimates. If this is not possible, you should somehow try to guess what these two objective probabilities in a two stage experiment really are. Only then can you interpret $P(X=\text{head})=0.58$ as your subjective believe about the long run frequency in a two stage experiment and can argue that you believe this leads to the best long run strategy. That being said, in my initial example, if I had to make a single bet, I personally would base my action on $p=0.6$, because I am not interested in the long run here. This does not mean $P(X=\text{head})=0.58$, which reflects the long run relative frequency if my prior also does.