My dataset is made of a label, $y_{t}$, which is the dependent variable, and about 20 columns of independent numeric variables, $X_{t}$, $t=1,2,...,T$.

These samples are time series and my goal is to classify $y_{t}$ according to $X_{t}$.

The dependent variable can get just two labels: "$0$" or "$1$".

Probability of belonging to "$0$" or "$1$" label is not needed, although it could bring further value to the analysis.

I would like to know which one of the following methods is the best one for my case and why (eventually how should I set methods and parameters if needed):

  • Support Vector Machines: which kernel should I use (linear, polynomial, radial basis, sigmoid)?
  • Neural Networks: how many layer and nodes should I set?
  • Random Forests
  • Non parametric model applied to binary outcome (this provides probabilities of belonging to each class)

What can you suggest me?


These decisions IMHO can only be made in a sensible way with intimate knowledge about the problem and the data at hand (search terms: no free lunch theorem for pattern recognition/classification). So all we can tell you here are very general rules of thumb.

  • The more statistically independent cases you have for training, the more complex models you can afford. Very restrictive models (e.g. linear) are very often chosen because more complex models cannot be afforded with the given amount of data and less about really being convinced of having actually linear class boundaries.
    See bias variance tradeoff and model complexity e.g. in The Elements of Statistical Learning

  • knowledge about the nature of your problem and data may also suggest sensible ways of feature generation.

  • If you don't have terribly many samples, but absolutely need nonlinear boundaries and therefore get unstable models, then ensemble models (like the random Forest) can help. You can aggregate not only decision trees but all other kinds of models as well.

  • There are rumours* that for the final quality of the model the choice of model often matters less than the experience the user has with the chosen type of model. I try to collect some evidence about this rumour in this question.
    The conclusion would be to look for someone to consult who has experience with the classifiers you consider or, even better, with classification of your type of data (that would need a more detailed description than just saying it is time series).

Note: the first three can also be set up to output posterior probabilities.

*I don't know any scientific study that reports this, but have heard numerous people reporting this observation, and there is a number of descriptions of the differences between types of models that at the end conclude that the theoretical differences in practice hardly ever matter.

  • $\begingroup$ The last one is a very good point. $\endgroup$ Jun 8 '13 at 17:16

It is hard to make specific suggestions without knowing more information about the exact problem you are attempting to solve. However, I will make a few recommendations for general time series prediction.

First, it is important to note that independence of observations is often a poor assumption for time series due to serial correlation of observations. Additionally, assuming that the time series is stationary may or may not be a good assumption. These issues complicate the partitioning of your data into training, cross-validation, and test sets. If there is a significant degree of autocorrelation this will reduce your effective sample size and increase your chances of over fitting, especially for complex machine learning algorithms like multilayer neural networks that have many tunable weights. Also, autocorrelation affects how you split your data set. If you randomly sample from your entire data set to create cross-validation and testing sets, then you will likely introduce a look-ahead bias into your out-of-sample error estimates. This happens because the out-of-sample data sets have points that are temporally adjacent to in-sample data points. This violates the independence assumption if there is significant autocorrelation and will artificially reduce your out-of-sample error estimates. This problem often leads to splitting the data set at a particular time point so that every observation before the split is training and every observation after the split is cross-validation/testing. This alleviates some of the issues with autocorrelation, but adds other complications if the in-sample time series differs significantly from the out-of-sample time series (non-stationary time series).

Considering the issues above, I would start as simple as possible and work my way up in complexity as needed. I would start by temporally splitting my data set into an in-sample and out-of-sample set, making sure that the out-of-sample contained similar cycles and statistical properties as the in-sample set. Then I would approach the in-sample data set with a Random Forest. Not knowing the size of your data set, 20 input variables may be a conservative use of your degrees of freedom. However, due to the negative effects of autocorrelation on your effective sample size, I would apply a random forest to the data set to get an estimate of variable importance. Knowing the relative importance of your inputs might help you eliminate some of your weaker inputs and free up more degrees of freedom. This site is a good intro for using random forests for variable importance estimates, and this paper discusses key pitfalls of conducting variable ranking. Random forests also have the advantage that you can pull out individual decision trees and understand the classification process by following the branches of the tree. This process of examining several of the forest’s decision trees can be very insightful and lead to better choices for the final classification algorithm. Neural networks and SVMs are less easy to interpret than random forests. Random forests also require less pre-processing of data than other machine learning algorithms, making them an approachable first step.

Using a random forest does have the drawback that it randomly splits data into out-of-bag (OOB) and in-bag samples for each decision tree. This random splitting will lead to bias in the OOB error estimate of the forest if your time series has significant autocorrelation. However, you can confine this bias to the in-sample data set by only running the random forest on it and preserving the out-of-sample data set for actual error estimation.

A random forest may provide you with enough power for your classification problem. However if you are looking for more improvement, I would proceed next by constructing a logistic regression using the variables deemed most important by the random forest. At this point, you have already used your in-sample data for variable selection and for insights by examining decision trees. However, you are probably safe splitting the in-sample data into training and cross-validation sets and then using learning curves to decide how complex you should make your classification algorithm. Start with a simple logistic regression, if you are still observing high bias via your learning curve, then you might consider moving to a multi-node, single layer neural network and repeating the process. Or you might consider using a SVM with a RBF kernel. This post touches on the differences between neural nets and SVMs. These choices will really be driven by the specifics of your problem and the availability of more data if you do encounter high variance in your final classification algorithm. Also, the complexity of your final classification algorithm will be governed by performance on cross-validation, but you should aim for the simplest solution that provides reasonable performance.

  • $\begingroup$ Autocorrelation is definitely in my time series sample. My sample is made of both the level and the first differences of time series, then there are both non stationary and (weakly) stationary variables. Does this make you to confirm what you have written above, that is, using random forests to analyse the sample then fitting a logistic regression on most important variables? $\endgroup$
    – Lisa Ann
    Jun 5 '13 at 14:35
  • 1
    $\begingroup$ Autocorrelation reduces your effective sample size and impacts the partitioning of your data. Knowing this I would gravitate towards simpler algorithms. Cbeleites' answer was good at highlighting the benefits of using ensemble methods like a random forest to accommodate nonlinearities with sample size constraints. Again, without knowing specifics about your problem it is difficult to make a recommendation, but I think a random forest would be a good starting place given that you are dealing with a binary classification and you have a large selection of input variables. $\endgroup$ Jun 5 '13 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.