# Is the observed allele frequency significantly less than the predicted?

Question: How can I construct a test to determine if the observed "mountain"-allele frequency (Fig 1) is significantly lower in the central to southern mountains than predicted (Fig 2) by the ecological selection model (see below for details)?

Problem: My initial thought was to regress the model residuals against latitude:longitude and altitude (which results in only the interaction between latitude and longitude as significant). The problem is that the residuals (Fig 3) may reflect variation not explained by the model and/or that their is something biological happening, e.g. allele has not had time to spread south to its potential or there is some barrier to gene flow. If you compare the observed (Fig1) versus expected (Fig 2) mountain-allele frequencies there is an obvious difference especially in the central to southern mountains of Sweden and Norway. I accept that the model may not explain all of the variation, but can I come up with a reasonable test to explore the idea that mountain-allele has not reached its potential in the central to southern mountains?

Background: I have an bi-allelic AFLP marker whose frequency distribution appears to be associated with mountain (and latitude:longitude) versus lowland habitats on the Scandinavian peninsula (Fig 1). The "mountain"-allele is nearly fixed in the north, which is mountainous. It is nearly absent or fixed for the "lowland"-allele in the south, which lacks mountains. As one moves north to south in the mountains the "mountain"-allele occurs in lower frequency. This difference in the "mountain"-allele frequency from north to south may simply be due to phylogeography or historical processes, as the region was colonized from both the north and the south. For example, if the mountain-allele originates in the northern population, maybe it has not had time to expand fully into the southern population, assuming there are no barriers to gene flow (the northern and southern populations are panmictic for mtDNA and 12 microsatellites).

My working hypothesis is that the "mountain"-allele frequency is a result of ecological selection (null hypothesis is neutral selection).

For my ecological selection model, I have used a generalized additive model (GAM) with the binomial allele frequency as the response variable (129 sites sampled across Fennoscandinavia with typically 10 to 20 individuals sampled at each site) and several climatic and growing season variables as the predictor variables. The model results are as follows (TMAX04-06 = max temperature in April-June, Phen_NPPMN = mean growing season vegetation productivity, PET_HE_YR = annual potential evapotranspiration, Dist_Coast = distance to coast):

Family: binomial

Formula: Binomial_WW1 ~ s(TMAX_04) + s(TMAX_05) + s(TMAX_06) + s(Phen_NPPMN) +
s(PET_HE_YR) + s(Dist_Coast)

Parametric coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.74372    0.04736   -15.7   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Approximate significance of smooth terms:
edf Ref.df Chi.sq  p-value
s(TMAX_04)    3.8100  4.812 25.729 9.43e-05 ***
s(TMAX_05)    0.8601  1.000  5.887  0.01526 *
s(TMAX_06)    0.8862  1.000  7.644  0.00569 **
s(Phen_NPPMN) 6.2177  7.375 39.028 3.16e-06 ***
s(PET_HE_YR)  3.1882  4.147 18.039  0.00145 **
s(Dist_Coast) 2.2882  2.857  9.725  0.01906 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

R-sq.(adj) =  0.909   Deviance explained = 89.7%
REML score = 326.73  Scale est. = 1         n = 129


Figure 1. Observed "mountain"-allele frequency for bi-allelic AFLP marker. Contour lines 0.1 frequency intervals, color-shading is altitude with blues for lowest and reds highest.

Figure 2. Predicted"mountain"-allele frequency for bi-allelic AFLP marker. Contour lines 0.1 frequency intervals, color-shading is altitude with blues for lowest and reds highest.

Figure 3. Ecological selection model (using GAM) residuals broken down by the entire study area (Fennoscandinavia) and separately for Norway, Sweden and Finland. The red dashed lines represent a secondary contact zone between the northern and southern populations inferred from other AFLP markers and stable isotope analysis of feathers grown on their separate wintering grounds in Africa. The thin black dotted line is the center of the zone.

• Thanks for the large amount of detail. How does the null model come into the question? It seems that only the ecological selection model matters in comparing observed vs expected. – gui11aume Jun 30 '12 at 22:46
• What is your actual data here? The model output says "n=129", but is that 129 individuals, or 129 locations at each of which you have allele frequencies, or ... ? – onestop Jul 1 '12 at 15:54
• Sorry I wanted to provide as much background as possible to allow you all to best understand the logic (or illogic) of my approach for the manuscript I am developing. My question stands at the top, I would like to find a why to determine if the difference between observed and expected is significantly different. This might inform me of whether the allele is still spreading or there is some barrier to reaching the expected value. As for the "N=129", as this is an allele frequency, this means 129 sites with typically between 10 and 20 individuals sampled at each site. – Keith Larson Jul 10 '12 at 14:26
• You mention that individuals appear to be panmictic for the maternal and automsomal loci. Do you have access to these data? How did you assess panmixia? – Sameer Oct 24 '12 at 23:46