1
$\begingroup$

I'm trying to predict when a location has enough wind to go sailing. I've built several models that look for correlations between different weather conditions and whether it ends up being windy X hours later. Ultimately, I want to produce something like "Based on current conditions, there is a 70% chance it will be windy 1 hour from now."

I have several models that consider different factors, so my challenge is figuring out which one to use.

In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between?

Is that the right way to compare models? How should I calculate an overall score for each model?

Thx in advance!

$\endgroup$
  • 1
    $\begingroup$ This question might be better suited to Data Science.SE.com, since it's not really about statistics per se. Consider flagging it for migration if you don't get good answers here. $\endgroup$ – Stephan Kolassa Aug 28 '14 at 7:06
2
$\begingroup$

There are several different ways to evaluate the performance of predictive models. Depending on your context you may need more or less sophistication.

The simplest measure is the concept of accuracy = (correct predictions) / total predictions. For this measure, higher accuracy is better.

Although very simple, accuracy is subject to several problems in particular it does not differentiate a false positive (you say that is windy and in fact it is not) from a false negative (when you say that it is not windy and it is).

If false positives and false negatives have different costs (or benefits depending on how you want to put it) for your scenario, then accuracy is not good enough for you.

In such cases, you should start with what is called a ROC analysis which will essentially show how different classifiers trade off true positives with false positives. This will allow you to get a better understanding of the performance of your model.

While you are at it, you should also look up the concept of confusion table which is essentially a contingency table of the output of your classifier, that is essential to build all sorts of evaluation metrics for predictive models.

An introduction to some key concepts is provided in this MIT presentation Evaluating Predictive Models.

Hope this helps,

$\endgroup$
  • $\begingroup$ I up-voted and agree with everything. In this case where it appears that there is a desire for risk assessment rather than classification I might emphasize a well calibrated model with Brier score/c-statistic/validation curve (ncbi.nlm.nih.gov/pmc/articles/PMC3575184) But these methods of assessment may not work well for non-likelihood based models (trees/tree ensembles etc) that the poster appears to be considering. $\endgroup$ – charles Sep 4 '14 at 3:17
1
$\begingroup$

If you see your problem as trying to calculate the probability of wind, it's no technically the same the problem of trying to predict when it will be windy or not.

In the first case your model will output "70%", "15%", "0%", etc. It's a regression problem, and a model can be evaluated using means squared error, for example.

In the second case you are trying to build a model that will output "yes" or "no", and it's a classification problem, and then the usual metrics are accuracy, f1 measure, AUC (area under ROC curve), etc.

In both cases you need to evaluate the model against a test set where you know the answers (usually you set aside some of the training data for this purpose). I would say in this case your problem is most likely a classification problem. as you say yourself. Now thinking of the weather reports that give probabilities of rain, that can perhaps be seen as a prediction + measure of the confidence. 80% chance of rain (wind... ) is kind of like predicting a yes with a pretty high confidence, 40% chance of rain is kind of predicting a "no" with a low confidence.

Some machine learning tools will allow you to put a confidence value on the result, others won't.

Finally I will just recommend the free open source tool Weka, it implements a whole bunch of different kinds of classifiers and will do the evaluation for you, too.

$\endgroup$
1
$\begingroup$

The question and your self-answer touch on a wide range of machine learning concepts. I will share my opinion on a couple of these.

In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between?

So your prior probability of being windy is 0.15. You are right in your statement that you should take this prior probability of the event happening when you are evaluating the performance of your classifier. You can give the Bayesian Information Reward (BIR) a go.

The intuition of BIR is as follows: a bettor is rewarded not just for identifying the ultimate winners and losers (0's and 1's), but more importantly for identifying the appropriate odds. In doing so, it compares all predictions with the prior probabilities.

Let's say you have a list of 10 Arsenal (football team in England) games with possible outcomes: Win or Lose. The formula for binary classification rewarding per game is:

enter image description here

where, p is your model's prediction for a particular Arsenal game, and p′ is the prior probability of Arsenal winning a game. The catch-point is: if I know beforehand that p′=0.6, and my predictor model produced p=0.6,even if its prediction was correct it is rewarded 0 since it is not conveying any new information.

After doing a little more research, I believe what I want is either a decision tree or a multi-variate classification engine.

I think a Markov network (aka Markov Random Field) would be more appropriate. The wind will not materialise of thin air but will be blowing towads your location from neighbouring ones in certain directions. So the probability of observing wind (e.g. Wind speed > 10 mph) at any square mile will be closely dependent on its neighbours (with ideally a time lag - depending on how far or close you define the neighbours).

$\endgroup$
-1
$\begingroup$

After doing a little more research, I believe what I want is either a decision tree or a multi-variate classification engine.

Basically each of my weather conditions (e.g. temperature) is simply a feature. The machine learning algorithm can figure out which features are most relevant and ultimately make a prediction of wind / no wind based on a new set of features (the current weather conditions).

These links were very helpful:

$\endgroup$
  • $\begingroup$ this doesn't address the issue of evaluating the final model, decision trees are generally poor predictors $\endgroup$ – charles Sep 4 '14 at 2:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.