Let me say thanks in advance.

I'm working with a set of data that contains reported coyote sightings. I use 2/3 of the data for model calibration along with an equal number of pseudo absences. I developed all possible models and ranked them according to their AIC weights. I chose the top models, who's weights summed to 75%, and created averaged model estimates for each of my parameter. Now I would like to test the accuracy of my model using the 1/3 of the data I held out. I assume I will need another equal amount of pseudo absences to include in the validation.

My problem is that I have averaged parameter estimates. All the software packages that I have come across have settings where you train your model using a specific data set and test it against a specific data set. However, I want to validate my model using averaged parameters, rather than the parameters calculated from running a simple regression.

I'm assuming I need to transform my outputs into percentages and choose an error threshold. Then I can produce a simple omission/commission report. But as this is for my Master's thesis, I want to be sure I'm using a respected and widely used method.

Can anyone point me in the right direction? My advisers are used to traditional validation methods so I'm doing a bit of outsourcing. Again, thanks for any and all advice.

  • 3
    $\begingroup$ Every single element of the strategy you've described is invalid. Please do some background work. $\endgroup$ – Frank Harrell Jul 24 '13 at 15:15
  • 1
    $\begingroup$ What do you mean your "weights summed to 75%?" 75% of what? And by "weights," do you mean the coefficients in your models or the AIC scores of the models? Also, I don't understand why you anticipate needing to rescale your model's output to percentages (again, percentages of what? Probability?). Mind elaborating on your reasoning here? I guess I don't entirely understand what you're working with. $\endgroup$ – David Marx Jul 24 '13 at 15:19
  • $\begingroup$ My results from the "all possible models" produced a few hundred models. For the entire list I calculated the AIC weights (the weight that that model is the best choice given the data). My adviser and I decided to choose the first 28 models, who's sum of AIC weights totaled .75.... I anticipate having to rescale the model output to make it a true binary output where: anything calculated as .5 or lower would be an estimated absence and everything higher than .5 is an estimated presence. $\endgroup$ – Stuart Wine Jul 24 '13 at 15:25
  • $\begingroup$ Frank... If you draw your attention to Burnham and Anderson 2002, "Model Selection and Multinomial Inference". My strategy is not only valid but widely use in spatial statistics. Keep in mind i'm a geographer, not a statistician. $\endgroup$ – Stuart Wine Jul 24 '13 at 15:30
  • 2
    $\begingroup$ Being a Geographer doesn't absolve you when doing bone-headed things with regards to statistical methodology. You'd do well to heed Frank's rebuke as he is an expert in this area. You claim Burnham & Anderson in support of your methodology. I am pretty much certain that in their discussion of the Model Averaging approach that B&A are clear that the models to be averaged are to be specified a priori from plausible scientific hypothesis, not chosen from the set of all possible models you can fit to your data. At least that was what Anderson says in his Primer book - my B&A is at home... $\endgroup$ – Reinstate Monica - G. Simpson Jul 24 '13 at 16:21

Instead of averaging over your coefficients to develop a single regression formula, you should pass the input data to all of your models separately and take the average result (i.e. treating your 28 models as an ensemble classifier). The chapter you referenced describes how to weight the output of each of your models, so models that you evaluated with higher scores get more weight but are still treated as an ensemble. This is extremely similar to a technique called Bayesian Model Averaging (BMA): the difference is that instead of estimating the likelihood of each model via Akaike Score, the likelihood of each model is calculated as a posterior probability. You may find the BMA literature interesting. Still, once you have your model, you should be able to evaluate it the same way you would evaluate any other model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.