The mean (average) is a descriptive statistic, just like the variance (standard deviation), skewness, or the coefficients of a linear regression, etc. They are just the mathematical results of applying an equation to the data sample(s). As such they do not require any assumption on the distribution. These observed statistics depend only on the sample, and not on the population from which the sample came, and hence do not care about the population’s distribution.
The only assumption for computing observed statistics is that your data is i.i.d. Independent (a value in your sample is not influenced by the other values: otherwise, e.g. a mean is not meaningful, as its value will depend on the first sample you took, and may not converge). And Identically Distributed (otherwise, you are literally taking the mean of apples and oranges; again, not a meaningful description of the data).
Now, when you are trying to make inferences about the population based on the sample (and its observed statistics), you may need to make assumptions about the population’s distribution. For example, if you not only compute the mean, but also a 95% CI around it (this will require normality of the sampling distribution of the mean), or CI’s around the regression coefficients (normality of the residuals – errors), etc. Same for deciding if 2 means are equal, or statistically significantly different (normality of the samples).
Note that there are some techniques which do not require any assumption about the parent population distribution (distribution free techniques), such as e.g. the bootstrap. There are also other techniques which rely on a given distribution, but that population distribution is guarantied by definition (e.g. binomial CI of a proportion: that proportion does, by definition follow a binomial distribution – no need to assume, or specify this).
(NOTE: I said earlier that observed statistics are valid regardless of the parent population’s distribution, as long as your sample is i.i.d. This is technically not quite true; that particular statistic needs to “exist”. E.g. the Cauchy distribution is such a pathological distributions for which the mean and variance are undefined; so trying to compute them is not meaningful. However, in practice -applied statistics-, I would not worry about such a situation, which you have almost no chance of encountering).