The Wilcoxon-Mann-Whitney test requires that two distributions are symmetrical How can I check this assumption by using a hypothesis test and how apply it in R? If this assumption is not met what test should I use rather than the Wilcoxon-Mann-Whitney test?
The Wilcoxon-Mann-Whitney test requires that two distributions are symmetrical
No, it doesn't require symmetry of both distributions.
(What makes you think this is necessary?)
It requires exchangeability of the ranks under H0 (and not under H1); the most typical way to get that would be if the two distributions had the same shape when H0 is true. They don't have to have the same shape when its false.
How can I check this assumption by using a hypothesis test
Even if it were to be a necessary assumption, testing it on the samples is not especially relevant, since you'd only require it under H0, which you have no good reason to think is true; indeed with an equality null, it's almost certainly false, so the assumption applies under a situation which the data don't typically tell you much about.
Edit: Let me give an explicit example.
Imagine for the sake of argument that I have two samples, one notionally a sort of 'control' group and one a sort of 'treatment' group. There's a claim that the treatment performed on the treatment group is highly valuable (has a substantive effect on the measure of interest) but we don't really think the treatment does anything at all.
Now imagine that each population has a beta distribution (under both H0 and H1). If the treatment does nothing, the parameters of the distribution would be the same (so that indeed the Wilcoxon-Mann-Whitney would have exchangeable ranks under H0).
If the treatment does something that corresponds to the claim we would expect the parameters to change.
For example, if the alternative is true we could see something like this:
while if the treatment did nothing at all, we would expect to see the black distribution for both populations.
If $H_1$ were true and we had samples from these two distributions, checking the assumption of the same shape under $H_0$ by looking at the data would be completely misleading $-$ in a reasonable-sized sample we would be highly likely to conclude that the shapes differ.
If we then conclude that there's a problem with the test, we are making a grave error. We're actually in a situation that the test is designed for!