Prediction accuracy between Random Variables I'm hoping to get some intuition behind the following problem:
Let's say I have 3 random variables: 


*

*$X$, that can take on the values c('A','B'), and does so with probability c(0.8, 0.2)

*$Y$, that can take on the values c('A','B'), and does so with probability c(0.8, 0.2)

*$Z$, that can take on the value c('A'), with probability c(1.0)
Obviously, if I were to make a the following prediction: $X$ = $Z$, I would be correct 80% of the time. 
If I predicted $Y$, (that has the same values and probabilities of $X$), then $X = Y$ 68% of the time (0.8*0.8 + 0.2*0.2)
The intuition I am trying to get at is: Even though $Y$ is a better representation of $X$ (as it is identically distributed), it is a less accurate prediction of $X$ than $Z$. Where is the flaw in my thinking? 
My intuition says that $Y$ has more information about $X$ than $Z$ does. But that does not pan out in prediction accuracy. 
 A: There are two issues here.
The first flaw is subtle.  When you use $Y$ as a prediction for $X$, you really want to use the distribution of $Y$ as the prediction, not the realizations you get by sampling from $Y$.  In this case the fix is rather simple, we structure our models to predict $P(X = A)$.  As you note, even if we have the correct distribution, two independent samples from it are rarely going to agree.  Imagine if we had a non-discrete outcome space (like for a normal distribution), even if we had it exactly correct, we would never get the same sample twice!
The second flaw is a common one: you are measuring the quality of your predictions with a metric that is not maximized by knowing the truth.
As you note, the most information you could have is knowing the distribution of $X$.  Any metric you design to measure the quality of your predictions should reflect this fact, it should be maximized by the prediction that $P(X = A) = 0.8$.  Metrics with this quality have a name, they are called proper scoring rules.
For example, the log-loss, used as he objective function in logistic regression, is such a metric.  It will be minimized (or maximized if you use the negative log-loss) by the forecast that $P(X = A) = 0.8$.
Classification accuracy is not a proper scoring rule, and is not appropriate for measuring the quality of probabilistic forecasts.  It's better to enforce a seperation of concerns that


*

*Models estimate the probability that events happen under certain conditions.

*Decision rules use these estimated probabilities, along with costs and benefits associated with the problem domain, to make decisions.

