What does entropy tell us? I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following: 

The probability distribution of the events, coupled with the
  information amount of every event, forms a random variable whose
  expected value is the average amount of information, or entropy,
  generated by this distribution.

So if I calculate the entropy associated with a probability distribution that is continuous, what is that really telling me?  They give an example about flipping coins, so the discrete case, but if there is an intuitive way to explain through an example like that in the continuous case, that would be great!
If it helps, the definition of entropy for a continuous random variable $X$ is the following: 
$$H(X)=-\int P(x)\log_b P(x)dx$$
where $P(x)$ is a probability distribution function.

To try and make this more concrete, consider the case of $X\sim \text{Gamma}(\alpha,\beta)$, then, according to Wikipedia, the entropy is
\begin{align}
H(X)&=\mathbb{E}[-\ln(P(X))]\\
&=\mathbb{E}[-\alpha\ln(\beta)+\ln(\Gamma(\alpha))+\ln(\Gamma(\alpha))-(\alpha-1)\ln(X)+\beta X]\\
&=\alpha-\ln(\beta)+\ln(\Gamma(\alpha))+(1-\alpha)\left(\frac{d}{d\alpha}\ln(\Gamma(\alpha))\right)
\end{align}
And so now we have calculated the entropy for a continuous distribution (the Gamma distribution) and so if I now evaluate that expression, $H(X)$, given $\alpha$ and $\beta$, what does that quantity actually tell me?  
 A: The entropy tells you how much uncertainty is in the system. Let's say you're looking for a cat, and you know that it's somewhere between your house and the neighbors, which is 1 mile away. Your kids tell you that the probability of a cat being on the distance $x$ from your house is described best by beta distribution $f(x;2,2)$. So a cat could be anywhere between 0 and 1, but more likely to be in the middle, i.e. $x_{max}=1/2$. 

Let's plug the beta distribution into your equation, then you get $H=-0.125$.
Next, you ask your wife and she tells you that the best distribution to describe her knowledge of your cat is the uniform distribution. If you plug it to your entropy equation, you get $H=0$. 
Both uniform and beta distributions let the cat be anywhere between 0 and 1 miles from your house, but there's more uncertainty in the uniform, because your wife has really no clue where the cat is hiding, while kids have some idea, they think it's more likely to be somewhere in the middle. That's why Beta's entropy is lower than Uniform's.

You might try other distributions, maybe your neighbor tells you the cat likes to be near either of the houses, so his beta distribution is with $\alpha=\beta=1/2$. Its $H$ must be lower than that of uniform again, because you get some idea about where to look for a cat. Guess whether your neighbor's information entropy is higher or lower than your kids'? I'd bet on kids any day on these matters.

UPDATE:
How does this work? One way to think of this is to start with a uniform distribution. If you agree that it's the one with the most uncertainty, then think of disturbing it. Let's look at the discrete case for simplicity. Take $\Delta p$ from one point and add it to another like follows: 
$$p_i'=p-\Delta p$$
$$p_j'=p+\Delta p$$
Now, let's see how the entropy changes:
$$H-H'=p_i\ln p_i-p_i\ln (p_i-\Delta p)+p_j\ln p_j-p_j\ln (p_j+\Delta p)$$
$$=p\ln p-p\ln [p(1-\Delta p/p)]+p\ln p-p\ln [p(1+\Delta p/p)]$$
$$=-\ln (1-\Delta p/p)-\ln (1+\Delta p/p)>0$$
This means that any disturbance from the uniform distribution reduces the entropy (uncertainty). To show the same in continuous case, I'd have to use calculus of variations or something along this line, but you'll get the same kind of result, in principle.
UPDATE 2:
The mean of $n$ uniform random variables is a random variable itself, and it's from Bates distribution. From CLT we know that this new random variable's variance shrinks as $n\to\infty$. So, uncertainty of its location must reduce with increase in $n$: we're more and more certain that a cat's in the middle. My next plot and MATLAB code shows how the entropy decreases from 0 for $n=1$ (uniform distribution) to $n=13$. I'm using distributions31 library here.

x = 0:0.01:1;
for k=1:5
    i = 1 + (k-1)*3;
    idx(k) = i;
    f = @(x)bates_pdf(x,i);
    funb=@(x)f(x).*log(f(x));
    fun = @(x)arrayfun(funb,x);
    h(k) = -integral(fun,0,1);
    subplot(1,5+1,k)

    plot(x,arrayfun(f,x))
    title(['Bates(x,' num2str(i) ')'])
    ylim([0 6])
end

subplot(1,5+1,5+1)
plot(idx,h)
title 'Entropy'

A: 
what does that quantity actually tell me?

I'd like to plug in a straightforward answer as follows: 
It's intuitive to illustrate that in a discrete scenario. Suppose that you toss a heavily biased coin, saying the probability of seeing head on each flip is 0.99. Every actual flip tells you very little information because you almost already know that it will be head. But when it comes to a fairer coin, it's harder for you to have any idea what to expect, then every flip tells you more information than any more biased coin. The quantity of information obtained by observing a single toss is equated with $\log \frac{1}{p(x)}$. 
What the quantity of the entropy tells us is the information every actual flipping on average(weighted by its probability of occuring) can convey: $E \log \frac{1}{p(x)} = \sum p(x) \log \frac{1}{p(x)} $. The fairer the coin the larger the entropy, and a completely fair coin will be maximally informative. 
A: I  don't feel that the most of the answers provided above are answering the question posed, except the comment by whuber. If I understand correctly, the original question pertains to DISCRETE cases as opposed the continuous cases. My impression is that RustyStatistician knows well what entropy means in discrete cases but is not sure of its meaning in continuous cases. Here is my answer: it does not mean much!
The following are my reasons:

*

*I have had the same question for years, have had searched for a satisfactory answer for years, but without success.

*To me one of the most important properties of entropy is its label invariance in discrete cases - it does not change its value under permutations on the index set $\{k;k\geq 1\}$, as in $\{p_{k}; k\geq 1\}$. As it is label-invariant, it measures internal volatility of a random element (as opposed to a random variable). In continuous cases (continuous random variables $X$), entropy changes its value if the values on the real line are arbitrarily exchanged. This fact puts entropy in continuous cases in a very different category. It does not mean that it is useless, but is of different meaning and different utility.

*I have consulted a world class expert on entropy, his best answer was: if you ask 10 experts the same question, they will give you at least 9 different answers.

*Legend has it that Kolmogorov once said to his students that entropy in continuous cases does not mean much. Many years later, I asked one of his students for a reference. He said with heavy Russian accent, "perhaps he just said it."

*One day Confucius was asked a question by one of his pupils, "what happens after one dies?" Confucius replied, "it is best to not answer this question until we figured out what happens before one dies."

