Entropy can be thought of as a non-parametric statistic about the spread of a distribution. Consider the follow unitless definition of entropy.
$$-\sum_n p_n\log_n p_n$$
If we define $0\log 0 = 0$, then the image of the function is $[0, 1]\forall n<\infty$.
Suppose that we have the uniform discrete distribution, with the pmf given by: $$1\over n$$
The value of our function for this distribution is 1 for all values of n.
Now consider the multinomial distribution given by: $$[ 0.01, 0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01]$$
The value of our function for this distribution is around 0.22.
Consider the multinomial given by: $$[ 0.001, 0.001, 0.001, 0.001, 0.001, 0.991, 0.001, 0.001, 0.001, 0.001]$$
The value of our function for this distribution is around 0.03.
So the more "concentrated" our distribution is, the more we know about the location of our realizations. It's easy to see by the definition of our function that it is invariant to translations of the location, or clustering of the high probability regions.
Regarding your questions, in the context of the linked paper.
- The uniform distribution has the largest entropy of any distribution. There is no information regarding the location of realized samples from this distribution because all locations are equally probable.
- If we imagine a dirac delta like distribution, where the mean equals the mode equals the max, equals the min, then the entropy is minimized. Minimization of the entropy of the distribution of residuals can be worded as gaining the most information about the location of the residuals. If all of the regression residuals are 1.5 then the entropy is minimized.
- See 2
One naggle with entropy and related information theory measures, is that the unit is defined by the base of the logarithm. $e$ is nats, $2$ is bits. This means that, without normalization, the support of the distribution influences the measure.