In a sense this is analogous to a situation where you test for differences in group means with ANOVA and then perform a post hoc test, such as Tukey's HSD, to tell which groups are the ones that actually differ. But, there is no equivalent post hoc test for Fisher's test.
The only "post hoc" thing that comes to mind is to run all pairwise comparisons for the table, and correct the p-values accordingly with, e.g., the Bonferroni method.
For a Chi square test, you could check the residuals or simply the expected-observed counts. In addition, going throught the percentages of observations in each group would probably answer your question at least partly, and this could be used with either Fisher's or Chi square test.
In R these can be done as follows:
# Percentages for rows and columns
# These a higher proportion of females than males in group D
prop.table(tab, 1) # rows
prop.table(tab, 2) # columns
# Chi square residuals
# The largest residuals are in the group D
chisq.test(tab)$residuals
# Chi square expected-observed
chisq.test(tab)$expected-chisq.test(tab)$observed
# Chi square "post hoc" test
# For Fisher you need to do this by hand
library(NCstats) # from rforge.net
chisqPostHoc(chisq.test(t(tab))) # for A-D
chisqPostHoc(chisq.test(tab)) # for gender