Question: How can I construct a test to determine if the observed "mountain"-allele frequency (Fig 1) is significantly lower in the central to southern mountains than predicted (Fig 2) by the ecological selection model (see below for details)?
Problem: My initial thought was to regress the model residuals against latitude:longitude and altitude (which results in only the interaction between latitude and longitude as significant). The problem is that the residuals (Fig 3) may reflect variation not explained by the model and/or that their is something biological happening, e.g. allele has not had time to spread south to its potential or there is some barrier to gene flow. If you compare the observed (Fig1) versus expected (Fig 2) mountain-allele frequencies there is an obvious difference especially in the central to southern mountains of Sweden and Norway. I accept that the model may not explain all of the variation, but can I come up with a reasonable test to explore the idea that mountain-allele has not reached its potential in the central to southern mountains?
Background: I have an bi-allelic AFLP marker whose frequency distribution appears to be associated with mountain (and latitude:longitude) versus lowland habitats on the Scandinavian peninsula (Fig 1). The "mountain"-allele is nearly fixed in the north, which is mountainous. It is nearly absent or fixed for the "lowland"-allele in the south, which lacks mountains. As one moves north to south in the mountains the "mountain"-allele occurs in lower frequency. This difference in the "mountain"-allele frequency from north to south may simply be due to phylogeography or historical processes, as the region was colonized from both the north and the south. For example, if the mountain-allele originates in the northern population, maybe it has not had time to expand fully into the southern population, assuming there are no barriers to gene flow (the northern and southern populations are panmictic for mtDNA and 12 microsatellites).
My working hypothesis is that the "mountain"-allele frequency is a result of ecological selection (null hypothesis is neutral selection).
For my ecological selection model, I have used a generalized additive model (GAM) with the binomial allele frequency as the response variable (129 sites sampled across Fennoscandinavia with typically 10 to 20 individuals sampled at each site) and several climatic and growing season variables as the predictor variables. The model results are as follows (TMAX04-06 = max temperature in April-June, Phen_NPPMN = mean growing season vegetation productivity, PET_HE_YR = annual potential evapotranspiration, Dist_Coast = distance to coast):
Family: binomial
Link function: logit
Formula: Binomial_WW1 ~ s(TMAX_04) + s(TMAX_05) + s(TMAX_06) + s(Phen_NPPMN) +
s(PET_HE_YR) + s(Dist_Coast)
Parametric coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.74372 0.04736 -15.7 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Approximate significance of smooth terms:
edf Ref.df Chi.sq p-value
s(TMAX_04) 3.8100 4.812 25.729 9.43e-05 ***
s(TMAX_05) 0.8601 1.000 5.887 0.01526 *
s(TMAX_06) 0.8862 1.000 7.644 0.00569 **
s(Phen_NPPMN) 6.2177 7.375 39.028 3.16e-06 ***
s(PET_HE_YR) 3.1882 4.147 18.039 0.00145 **
s(Dist_Coast) 2.2882 2.857 9.725 0.01906 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
R-sq.(adj) = 0.909 Deviance explained = 89.7%
REML score = 326.73 Scale est. = 1 n = 129

Figure 1. Observed "mountain"-allele frequency for bi-allelic AFLP marker. Contour lines 0.1 frequency intervals, color-shading is altitude with blues for lowest and reds highest.

Figure 2. Predicted"mountain"-allele frequency for bi-allelic AFLP marker. Contour lines 0.1 frequency intervals, color-shading is altitude with blues for lowest and reds highest.

Figure 3. Ecological selection model (using GAM) residuals broken down by the entire study area (Fennoscandinavia) and separately for Norway, Sweden and Finland. The red dashed lines represent a secondary contact zone between the northern and southern populations inferred from other AFLP markers and stable isotope analysis of feathers grown on their separate wintering grounds in Africa. The thin black dotted line is the center of the zone.