I found two very useful posts about the difference between linear regression analysis and ANOVA and how to visualise them:
As stated in the first post, to test whether the average height of male and females is the same you can use a regression model ($y = \alpha + \beta x + \epsilon$, where $y$ denotes height and $x$ denotes gender) and test whether $\beta = 0$. If $\beta = 0$, then there is no difference in the height between males and females. However, I am not quite sure how this is tested when you have three groups. Imagine the following example:
height (y) - group (x) 5 - A 6 - A 7 - A 6 - A 30 - B 32 - B 34 - B 33 - B 20 - C 19 - C 21 - C 22 - C
The regression model would look like:
$$y = a+ b x + \epsilon$$
I quickly visualized the data (see image below)
They way I understood the regression model is that it would now test whether any of the three slopes (AB, AC or BC) has a slope $b$ which is significantly different from 0. If that's the case one can conclude like in an ANOVA that there is at least one group in which height is significantly different from one or more groups. Afterwards, one could use a post-hoc test of course to test which of the groups really differ. Is my understanding of how the regression models tests this hypothesis correct?