Relationship between entropy and predictability The entropy of a random variable $X$ is defined as $\mathbb{E}(-\log(f(X)))$ (where $f$ is the pdf of X, https://en.wikipedia.org/wiki/Entropy_(information_theory)).
Is there any general relationship between the entropy of $X$ and the possibility to "more or less accurately" predict the value of $X$ using some constant $a$? That is, are there known general relationships between $\mathbb{E}(\log(X))$ and measures such as
$$ \min_a \mathbb{E}[|X-a|] \tag 1$$
$$ \min_a \mathbb{E}[(X-a)^2] \tag 2$$
$$ \min_a \mathbb{E}[I(X=a)] \tag 3$$
?
For a $Be(p)$, for example, it seems that the two kinds of metrics (entropy on the one hand, and either of (1), (2), or (3) on the other) move jointly with $p$ (i.e., if $Be(p)$ has higher entropy than $Be(p')$, then $Be(p)$ is also less accurately predictable --- higher minimum expected error --- than $Be(p')$ in terms of (1), (2), or (3)).
Is something like that true more generally? For example, is it true in general that an increase in entropy implies a decrease in predictability as measured by (1), (2), or (3)? If it's not true in general, is true for some class of random variables larger than the class of $Be(p)$?
 A: I think there are counter-examples for all three measures. That is, for all three measures, there are cases where the entropy remains unchanged but the measure changes. Below are my proposed counter-examples.
Mean absolute distance (1)
Note that
$$\min_a \mathbb{E}[|X-a|] = \mathbb{E}[|X-\text{med}(X)|].$$
Let $f_Y(1) = f_Y(2) = f_Y(3) = 1/3$ and $f_Z(0) = f_Z(2) = f_Z(4) = 1/3$.
We have $\text{med}(Y) = \text{med}(Z) = 2$ and $H(Y) = H(Z) = -\log(1/3)$, though
$$ \mathbb{E}[|Z-\text{med}(Z)|] = 4/3 > 2/3 =  \mathbb{E}[|Y-\text{med}(Y)|].$$
Mean squared distance (2)
Note that
$$\min_a \mathbb{E}[(X-a)^2] = \mathbb{E}[(X-\mathbb{E}(X))^2] = Var(X).$$
Let $Y = Be(p)$ and $Z = 2*Be(p)$. Note that the two RVs have the same entropy:
$$ \begin{align} H(Y) && = && f(0) * [-\log(f(0))] && + && f(1) * [-\log(1-f(1))] \\ &&=&& p* [-\log(p)] && + && (1-p)*[-\log(1-p)]. \end{align}$$
$$ \begin{align} H(Z) && = && f(0) * [-\log(f(0))] && + && f(2) * [-\log(1-f(2))]\\ && =&& p* [-\log(p)] && +&&  (1-p)*[-\log(1-p)]. \end{align}$$
Of course, the two RVs have different variances with $Var(Z) = 4*Var(Y)$.
This seems to hold for any discrete RV, that is, if $Z = a + bY$, the two RVs have different variances ($Var(Z) = a^2 Var(Y)$), but have the same entropy.
As noted in the answer @whuber points at in the comments (How does entropy depend on location and scale?), this is not true for continuous distributions ("scaling a continuous variable [by $\sigma$] (which, for $\sigma \geq 1$ "stretches" or "smears" it out) increases its entropy by $\log(\sigma)$.").
Mean error rate (3)
Note that
$$\min_a \mathbb{E}[I(X=a)|] = \mathbb{E}[I(X=\text{mode}(X))].$$
Let $f_Y(1) = 1/8$, $f_Y(2) = 1/2$, $f_Y(3) = 3/8$ and $f_Z(1) = 1/4$, $f_Z(2) = 1/2$, $f_Z(3) = 1/4$.
We have $\mathbb{E}[I(Y=\text{mode}(Y))] = 1/2 = \mathbb{E}[I(Z=\text{mode}(Z))]$. However, $H(Y) \neq H(Z)$.
