If the answer to any of the above questions is yes, then what about (say) Lévy distribution where the mean and the variance are infinite, while the median and mode are functions of scale parameter? Am I mixing something here?
The answer to both questions is yes. If (and that's the important part) the respective quantities exist. For the location parameter, it is part of the definition and the given quantities (mean, etc.) are just examples which fulfill the definition.
For the variance it is more specific, since it is just a single example; even for all practical and most theoretical purposes the most important.
But as in so many cases, if the value is undefined, it can't be any of the given parameter types. There are even more common examples Levy, like Cauchy/Lorentzian. Another unfortunate side-effect is then, that the central limit theorem does not apply, so mixing these distribution with other, more well-behaved distributions has ill side-effects.
