What machine learning model would I use for predicting rainfall given numerical data

Question

I'm brand new to machine learning and am currently taking a course on it. In the course, we have been using things such as gradient boosting, and linear regression to take data that has been collected and use it to predict the amount of rainfall at a particular location.

This data, X has been given to us to use, and consists of 60,000 rows and 91 columns. Each of these 91 columns has something to do with the amount of rainfall (at least from my understanding). These columns have to do with the date, time of year, cloud patches, pressure at the time, velocity of wind, infrared data containing water vapor amounts, humidity, and brightness, and much more satellite and other types of data. This data gets mapped to a corresponding Y value which is the amount of rainfall given that specific row of data.

Each time we use this data (we use kaggle), we have to take the model we trained on and predict rainfall given a bunch of Xtest data. We make predictions and end up getting a mean squared error with a "score" back. Since I'm trying to learn more then just linear regression and gradient boosting (and have been encouraged to do so by my professor), I've been looking through some different models for machine learning. My question is, are any of these valid, and which model sounds right for the job?

1. kNN regression - Since I have a lot of fields of data, I figured running a kNN regression model would work well, using something like 50 nearest neighbors. The only problem I'm thinking of is how to translate 91 columns of data into a kNN regression problem. Would I have to essentially create one kNN model for each column and somehow reduce the dimensions until I have a 2D model that I can predict rainfall on? If so how would I accomplish something like this? Or am I completely wrong about kNN?

2. Deep learning/neural networks. I'll be honest I don't really understand a lot about the hidden layers, and looking through PyLearn2 (which my professor suggested) almost seems like a black box in a way. I can't really understand what is happening, and am not really learning much from it. From what I've read online, it also seems like I do not have enough training data for a good neural network. Despite that, would this be a recommended way to go about solving my problem?

3. Gradient boosting. So far, out of the two things I've tried (gradient boosting and linear regression), gradient boosting has yielded the best results, but not by much. Is there something I should be looking for to optimize an algorithm with this much data? Or is this not a good way to go?

I'm assuming linear regression is probably not the right job for this problem as I've had pretty bad results from using it.

However, I'm wondering, is there some model that could fit this problem? Or something I can learn about that might help me better understand a way to predict the rainfall?

Answer 1

1 kNN suffers a lot from the curse of dimensionality, and with 90 features they are very likely to perform badly, the underlying reason is that in a high-dimensional space a "neighbourhood" of a point is not anymore local and so you will consider points quite far from your target. (Refer to page 22 of Elements of Statistical Learning for detailed informations)

2 Being very brief, in regression context and with just one hidden layers (that usually is already enough), if M is the number of your hidden units, neural networks take M linear combinations of your features, apply a non-linear function to this combination to obtain the M hidden units, after this step they apply another linear combination of the hidden layers to get your output. M is determined by cross-validation. They are known to be a black-box, but if your only goal is prediction they behave quite well as they are able to capture non-linear relationships. However gradient boosting are probably equally not interpretable.

3 Gradient boosting are one of the state-of-art techniques and they generally require little pre-processing of the features as they are able to handle missing-values as well. The only effort to do is to cross-validate in order to obtain the best value for the shrinkage parameter and the number of trees, potentially the depth of a tree as well.

There is no ready recipe in order to determine which technique will perform better, the usual way to go is to try different models and cross-validate them in order to determine the optimal parameters and then choose the one with the best MSE.

Regarding the other techniques, you can try a GAM (generalized additive model). Explaining GAM here is probably hard, so refer to page 295 of Elements of Statistical Learning for more informations. They are known to have a good predictive power and still retain interpretability as you can see the single effect of every features.

If interpretation is not your goal, you can try Random Forests as well as they generally give good performance (probably they are more suited to a classification context, but it is worth trying).

Regarding the final question, even I'm far from being an expert in rainfall, I would expect some features to be highly correlated and so applying a principal components to reduce the dimension of the problem will probably help.