I have tried to find answers to my question by googling but haven't found anything relevant - if anyone could pass on some resources or relevant key words that would be brilliant, in absence of actually getting an answer to my question.
My model predicts deforestation at pixel resolution using both pixel- and lower-resolution predictors. The formula is as follows:
gam(formula = loss ~ ecozone + s(dist_river) + s(dist_prevdf) +
s(allarea) + s(dist_road) + s(altitude) + s(slope) + s(aspect,
bs = "cc") + s(pcARA, k = 3) + s(popdens) + s(pcpriorloss) +
s(pcforestGFC), family = binomial, data = gfcpix, weights = weight/mean(weight),
method = "REML", select = TRUE, knots = list(aspect = c(0,
aspect is an angle and therefore given a cyclic spline, and the
weights argument refers to loss having been oversampled by a factor of 5 in the data as it was very rare to begin with. However, as the maximum pixels sampled at the unit of the lower-resolution predictors was constrained, no-loss pixels were variably undersampled. Weights therefore differ between loss and no loss pixels across communities. Instead of including communities as random effects, as our main question relates to a few predictors at this resolution, I therefore added these predictors as-is. The total number of data rows in the model is 123288.
I used Faraway's logistic model instructions for evaluating the model and grouped predicted probabilities into intervals, comparing these groups into the corresponding proportion of
1 in the data. I multiplied probabilities by 5 to correspond to the sample rather than population level of
I am unsure why the model overpredicts at high probabilities. Plotting binned residuals against individual predictors did not suggest any clear issues, but I assume this could have been one way of chasing the issue? I attempted adding some plausible interaction structures into the model but they were not significant nor did they impact on the predictions. Of course, I may be missing some important predictors that would help in constraining the predictions. However, I am at a loss in terms of how to troubleshoot this issue. What might be the most likely cause of the problem, and how could I test this plus other possible causes? Simply saying "The model overpredicted loss at high probabilities" seems inadequate.
Any help would be much appreciated.
I have not provided data or code in this case as my question is more generic than my example, however can try to generate a similarly 'flawed' model with invented data if necessary.
UPDATE: To clarify the situation and partially respond to suggestions in previous comments:
I've plotted binned partial residuals as well for reference. I'm no expert on interpreting logistic model binned residuals but again following Faraway's book, I would say there is nothing SERIOUS in here (looks mostly like outlier departures rather than systematic issues?).
I redrew the predicted vs observed bins for various versions of the model, and it did seem that something was awry with the weights argument. Varying this argument alone by switching it off changed the model from having the bias to not having the bias. Playing with using the
logitlink also had a slight effect (when keeping weights as is).
In a way it is a solution that the
weightsargument is causing problems, but as I don't understand WHY I've not shut this case as resolved. High variability and low mean weights (0.2 value for all loss pixels, variable values >=1 for no loss pixels) characterises the area of bias, so can I assume that the low weights and fewer observations at high probability together combine to make observations in this range count less in the model fitting, allowing for a wobble?