Can modes be defined without assuming the existence of density function Surprisingly I have not found any literature. I would like to know if it is possible to define a (generalized) mode for general probability distributions, possibly defined on an infinitely dimensional space.
My thought is that it should be related to $ P(B_\theta(r))$, where $B_\theta(r)$ is a ball centered at $\theta$ with radius $r$.
 A: I guess that practical solution to your problem is simply to use kernel density estimation, where the mode is highest point of the estimated kernel density. It uses kernels $K_\theta$ centered at $\theta$ instead of $B_\theta$. You can easily find a number of papers describing this approach.
Another practical comment is that due to the curse of dimensionality you wouldn't be able to compute such mode and wouldn't be able to obtain meaningful KDE estimates as the number of dimensions grow.
A: With a continuous distribution mode is usually defined as the point with maximum probability distribution function (fdp). This is closely related to your thougth if it's taken to the limit - that is, for an infinitely small ball.
If $\text{vol}$ means volume (in whichever dimensions), one possible definition of fdp is:
$$fdp(\theta)=\lim_{r\to0}\frac{P(B_\theta(r))}{\text{vol}(B_\theta(r))}$$
Then, one possible definition of mode for a continuous distribution is the point where $P(B_\theta(r))$ per volume unit is maximum.
Furthermore, for certain applications a naive but useful way to estimate the mode from a sample is to fix a ball radius and find the centre of the ball that contains more sample points. For example, geologists used to plot joint orientations in stereographic projection to find the orientations where most points clustered, and therefore estimate the most likely ways a rock slope could fall. 
