Evaluating performance of machine learning models

Question

I'm trying to predict when a location has enough wind to go sailing. I've built several models that look for correlations between different weather conditions and whether it ends up being windy X hours later. Ultimately, I want to produce something like "Based on current conditions, there is a 70% chance it will be windy 1 hour from now."

I have several models that consider different factors, so my challenge is figuring out which one to use.

In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between?

Is that the right way to compare models? How should I calculate an overall score for each model?

Thx in advance!

Answer 1

There are several different ways to evaluate the performance of predictive models. Depending on your context you may need more or less sophistication.

The simplest measure is the concept of accuracy = (correct predictions) / total predictions. For this measure, higher accuracy is better.

Although very simple, accuracy is subject to several problems in particular it does not differentiate a false positive (you say that is windy and in fact it is not) from a false negative (when you say that it is not windy and it is).

If false positives and false negatives have different costs (or benefits depending on how you want to put it) for your scenario, then accuracy is not good enough for you.

In such cases, you should start with what is called a ROC analysis which will essentially show how different classifiers trade off true positives with false positives. This will allow you to get a better understanding of the performance of your model.

While you are at it, you should also look up the concept of confusion table which is essentially a contingency table of the output of your classifier, that is essential to build all sorts of evaluation metrics for predictive models.

An introduction to some key concepts is provided in this MIT presentation Evaluating Predictive Models.

Hope this helps,

Answer 2

If you see your problem as trying to calculate the probability of wind, it's no technically the same the problem of trying to predict when it will be windy or not.

In the first case your model will output "70%", "15%", "0%", etc. It's a regression problem, and a model can be evaluated using means squared error, for example.

In the second case you are trying to build a model that will output "yes" or "no", and it's a classification problem, and then the usual metrics are accuracy, f1 measure, AUC (area under ROC curve), etc.

In both cases you need to evaluate the model against a test set where you know the answers (usually you set aside some of the training data for this purpose). I would say in this case your problem is most likely a classification problem. as you say yourself. Now thinking of the weather reports that give probabilities of rain, that can perhaps be seen as a prediction + measure of the confidence. 80% chance of rain (wind... ) is kind of like predicting a yes with a pretty high confidence, 40% chance of rain is kind of predicting a "no" with a low confidence.

Some machine learning tools will allow you to put a confidence value on the result, others won't.

Finally I will just recommend the free open source tool Weka, it implements a whole bunch of different kinds of classifiers and will do the evaluation for you, too.

Answer 3

The question and your self-answer touch on a wide range of machine learning concepts. I will share my opinion on a couple of these.

In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between?

So your prior probability of being windy is 0.15. You are right in your statement that you should take this prior probability of the event happening when you are evaluating the performance of your classifier. You can give the Bayesian Information Reward (BIR) a go.

The intuition of BIR is as follows: a bettor is rewarded not just for identifying the ultimate winners and losers (0's and 1's), but more importantly for identifying the appropriate odds. In doing so, it compares all predictions with the prior probabilities.

Let's say you have a list of 10 Arsenal (football team in England) games with possible outcomes: Win or Lose. The formula for binary classification rewarding per game is:

where, p is your model's prediction for a particular Arsenal game, and p′ is the prior probability of Arsenal winning a game. The catch-point is: if I know beforehand that p′=0.6, and my predictor model produced p=0.6,even if its prediction was correct it is rewarded 0 since it is not conveying any new information.

After doing a little more research, I believe what I want is either a decision tree or a multi-variate classification engine.

I think a Markov network (aka Markov Random Field) would be more appropriate. The wind will not materialise of thin air but will be blowing towads your location from neighbouring ones in certain directions. So the probability of observing wind (e.g. Wind speed > 10 mph) at any square mile will be closely dependent on its neighbours (with ideally a time lag - depending on how far or close you define the neighbours).

Answer 4

After doing a little more research, I believe what I want is either a decision tree or a multi-variate classification engine.

Basically each of my weather conditions (e.g. temperature) is simply a feature. The machine learning algorithm can figure out which features are most relevant and ultimately make a prediction of wind / no wind based on a new set of features (the current weather conditions).

These links were very helpful:

• https://www.igvita.com/2007/04/16/decision-tree-learning-in-ruby

• https://github.com/mperham/bayes_motel