I'm trying to predict when a location has enough wind to go sailing. I've built several models that look for correlations between different weather conditions and whether it ends up being windy X hours later. Ultimately, I want to produce something like "Based on current conditions, there is a 70% chance it will be windy 1 hour from now."
I have several models that consider different factors, so my challenge is figuring out which one to use.
In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between?
Is that the right way to compare models? How should I calculate an overall score for each model?
Thx in advance!