Can anyone give me more details why is it always better to work with the distribution of a variable than with its average? I know that by taking the average we take into account the outliers but is it that bad? I mean it's quite easier to work with the mean... Can someone give me some examples why is it REALLY better to work with distributions.
It isn't "always" better to work with the full distribution. It's a matter of judgment as to what point it is reasonable to talk about the "average" value. The average is one particular characterization of the distribution, and how relevant it is depends on the situation. If you're running a casino, the average payout of your slot machines is highly relevant. On the other hand, if you're building a levee, building it as high as the average river height is a really bad idea.
One thing to keep in mind is that functions do not generally commute with averaging. That is, average$(f(x))$ is not in general the same as $f($average$(x))$. For instance, if you have a bunch of spheres, the formula $V=4\pi r^3/3$ doesn't work if you plug in the average radius; the average volume is not $4\pi/3$ times the average radius cubed. So often one needs to do analysis on the whole distribution, and then possibly take the average or some other summary statistic at the very end.
The average (mean) $\mu_1$ is the first moment of the distribution $f(x)$: $$\mu_1=\int xf(x)dx$$ There's an infinite number of moments $\mu_k$: $$\mu_k=\int x^kf(x)dx$$
Knowing the distribution includes knowing the average. Since the argument is that it's better to know more than less, we're told to better know the whole distribution than just its mean (average)
It isn't always better, it depends what you are trying to do. But taking just the average can lead you seriously astray.
However, here is a quote from Sir Francis Galton that make some points:
It is difficult to understand why statisticians commonly limit their inquiries to Averages, and do not revel in more comprehensive views. Their souls seem as dull to the charm of variety as that of the native of one of our flat English counties, whose retrospect of Switzerland was that, if its mountains could be thrown into its lakes, two nuisances would be got rid of at once. An Average is but a solitary fact, whereas if a single other fact be added to it, an entire Normal Scheme, which nearly corresponds to the observed one, starts potentially into existence. Some people hate the very name of statistics, but I find them full of beauty and interest. Whenever they are not brutalised, but delicately handled by the higher methods, and are warily interpreted, their power of dealing with complicated phenomena is extraordinary. They are the only tools by which an opening can be cut through the formidable thicket of difficulties that bars the path of those who pursue the Science of man.