Entropy can be developed as expected surprise, as I tried in Statistical interpretation of Maximum Entropy Distribution. I will work for the discrete case now, but most can be carried over to the continuous case.
Define a surprise function $\sigma \colon p \mapsto \sigma(p)$ which sends a probability to the surprise value of an event having that probability. As you gets less surprised by a more probable event, $\sigma$ should be decreasing, and $\sigma(1)=0$ as you are not surprised at all by a certain event occurring. The surprise function $\log\left( \frac1p \right)$ leads to Shannon entropy.
First, lets connect this to the example in the other answer by @Richard Hardy. Denote the discrete values by $x_i$ and suppose they are real numbers. Suppose there is one $x_m =\mu$, the expected value, and that $p(x_i)\leq p(x_m)$, and if $(x_i-\mu)^2 \geq (x_j-\mu)^2$ then so $p_i \leq p_j$. In this case
$$ \sigma \colon x\mapsto (x-\mu)^2$$ is a surprise function and the corresponding expected surprise is the variance. This way we can consider the variance a kind of entropy!
Back to generalities. A family of surprise functions is
$$ \sigma_\alpha \colon [0,1]\mapsto [0, \infty]; \quad
\sigma_\alpha(p)=\begin{cases} (1-\alpha)^{-1} (1-p^{\alpha-1})&, \alpha\not=1 \\
-\log p &, \alpha=1 \end{cases}$$
The expected surprise becomes
$$ D_\alpha(p_1, \dotsc, p_n)=\sum_i p_i \sigma_\alpha(p_i) = \\
\begin{cases} (\alpha-1)^{-1} (\left( 1-\sum_i p_i^\alpha\right) &, \alpha\not=1 \\ -\sum_i p_i\log p_i &, \alpha=1 \end{cases} $$
and we have used the name $D$ because in ecology this is known as diversity (as in biodiversity.) In ecology one often presents this in another way using the concept of effective number of species. The idea is that an ecosystem with $n$ species is most diverse if the frequency of all the species are the same, so $p_i=1/n$. In other cases we can calculate some $\text{effective number of species }\leq n$. I wrote about that here: How is the Herfindahl-Hirschman index different from entropy? so will not repeat. In the case of the Shannon entropy the effective number of species is given by the exponential of the entropy. Now write $A=\{p_1, \dotsc, p_n\}$ and
$$ \lvert A \rvert = e^{H(A)} =\prod_i p_i^{-p_i} $$ and call this the cardinality of $A$, to have a mathematical name useful also outside ecology. Think of this as a measure of the size of $A$. Now we want to extend this for all the surprise functions $\sigma_\alpha$. The result is (for the moment I jump the development)
$$\lvert A \rvert_\alpha = \begin{cases} \left( \sum_i p_i^\alpha\right)^{\frac1{1-\alpha}}&,\alpha\not=1 \\
\prod_i p_i^{-p_i}&, \alpha=1 \end{cases} $$
Now we can get back to the entropy scale by taking logarithms, and so we define the $\alpha$-entropy by $H_\alpha(A)=\log \lvert A \rvert_\alpha$. This is usually called the Renyi-entropy, and has better mathematical properties than the $\alpha$-diversity. All of this and more can be found starting here.
The measures we have discussed so far only use the probabilities $p_i$, so we did not answer the question yet---so some patience! First we need a new concept:
Cardinality of metric spaces Let $A$ be a set of points $a_1, \dotsc, a_n$ with given distances $d_{ij}$ ($d_{ij}=\infty$ is permitted.) Think of this as a finite metric space, but it is not clear we really need all the metric space axioms. Define a matrix $Z=\left( e^{-d_{ij}}\right)_{i,j}$ and a vector $w$ as any solution of $Z w = \left(\begin{smallmatrix}1\\ \vdots \\1 \end{smallmatrix}\right)$. $w$ is called a weighting of $A$. Now we can define the cardinality of $A$ as the sum of the components of $w$,
$$
\lvert A \rvert_\text{MS} =\sum_i w_i $$
It is an exercise to show this does not depend on the choice of $w$. Now we want to extend this definition to a ...
Metric probability space $A=(p_1, \dotsc, p_n; d)$ where $d$ is a distance function, a metric. To each point $i$ we associate a density $\sum_j p_j e^{-d_{ij}}$. Since $e^{-d_{ij}}$ is antimonotone in the distance $d$, it represents a closeness, so the density can be seen as an expected closeness around point $i$, which explains the terminology density. Define a similarity matrix $Z=\left( e^{-d_{ij}}\right)_{i,j}$ and probability vector $p=(p_1, \dotsc, p_n)$. Now $Zp$ is the density vector. For example, if all distances $d_{ij}=\infty$ then $Z=I$, the identity matrix, so $Zp=p$.
Now we will generalize by replacing in many formulas $p$ with $Zp$.
Earlier surprise only depended upon the probabilities of the observed event. Now we will also take into account the probabilities of nearby points. For instance , you will probably be very surprised by a python snake at Manhattan, but now we will measure that surprise also taking into account the probabilities of other snakes ... with the surprise function $\sigma$, the expected surprise is now defined as $\sum_i p_i \sigma\left( (Zp)_i\right)$ for a discrete metric space with all $d_{ij}=\infty$, this is no change.
Diversity is now generalized to
$$ D_\alpha(A)=\sum_i p_i \sigma_\alpha\left( (Zp)_i\right)=
\begin{cases} (\alpha-1)^{-1} \left(1-\sum_i p_i(Zp)_i^{\alpha-1} \right)&,\alpha\not=1 \\ -\sum_i p_i \log\left( (Zp)_i\right) &, \alpha=1\end{cases}
$$ For example, with $\alpha=2$, $D_2(A)= p^T \Delta p$, $\Delta=\left( 1-e^{-d_{ij}}\right)_{i,j}$ is known as Rao's quadratic diversity index, or Rao's quadratic entropy.
$\alpha$-Cardinality Correspondingly we have
$$ \lvert A\rvert_{\alpha} = \frac1{\sigma_\alpha^{-1}(D_\alpha(A))}=
\begin{cases} \left( \sum_i p_i (Zp)_i^{\alpha-1} \right)^{\frac1{1-\alpha}}&,\alpha\not=1 \\
\prod_i (Zp)_i^{-p_i} &, \alpha=1
\end{cases}
$$
and now the ...
$\alpha$-entropy is obtained by taking the logarithms of the $\alpha$-cardinality, and this way we now have obtained an entropy where distances between the points plays a role. All this and much more can be found here at the n-Category cafe. This is still relatively new theory, so new developments can be expected. The ideas comes originally from theoretical ecologists.
