I've learned of the existence of things like Monte Carlo Markov Chains, Neural Nets, and Random Forests. In school I learned about OLS, logit, probit, GLM, etc. While I learned those methods I learned everything in the context of establishing marginal effects with very little emphasis on predicting for the sake of predicting. This is obviously in stark contrast to rf and nn which (I believe) are specifically designed for making good predictions rather than solving for marginal effects. I'm not really sure what MCMC is best suited to solve for, I just know that it does magic by smashing a bunch of random things together to get a good real answer.

Assuming your main motivation is to predict future outcomes could anyone give me a quick (or relatively so) examples of when each method is better suited than the others? (Is this even a reasonable question?) Is OLS (and by extension all the methods where modeling upfront is rigid) ever likely to be the most accurate means to predict assuming you just care about predictions? Ignoring OLS, are there any pointers for choosing between the various machine learning techniques when trying to derive a model to make predictions or is it best to run all the techniques and see how they do for a given problem/set of data?

  • 1
    $\begingroup$ This made laugh pretty hard: "I'm not really sure what MCMC is best suited to solve for, I just know that it does magic by smashing a bunch of random things together to get a good real answer." Too perfectly stated. $\endgroup$
    – Taal
    Oct 2, 2013 at 22:26

1 Answer 1


This is an interesting question - in a lot of ways, it's really the question in Machine Learning.

The short version of the answer is something like this: your choice of model is very important, but there are no silver bullets or magic hammers to be had; with a few notable exceptions, it's worth thinking of all of the techniques that you've learned as tools to be used in the right situation. The actual choice that takes place usually involves some analysis, some visualization, some intuition, some experience, and a little bit of voodoo (which is to say, luck).

I'll answer you in two parts: One, I'll give examples of problems suited to particular models, and two, I'll outline some approaches that I take when I'm first setting things up.

Consider the problem of image recognition in a complex environment. It's not an easy problem to solve; it can often deal with rich features and different levels of abstraction. In this case, one might try a neural network of some variety(let's say something fancy, like a convolutional net): this has the advantage of being a very expressive model, which might be just the power tools needed to model the complex interrelationships of the variables. Of course, the penalty paid is that the algorithm can be rather expensive; however, it might be that this slowness is justified given the advantage conferred by the richness of the model.

Now, consider the very different problem of finding the general trend in noisy time-series data over a long span of time; let's say our choices are "going up" or "going down". In this case, one might be tempted to use a very complicated solution like the neural net above; as we mentioned, it would be a powerful tool, which could explain complicated relationships in the data. On the other hand, for a question with such a limited scope, it might be useful to simply use OLS regression and look at the slope of a linear equation fit to the data. It would be quick and dirty, but it would probably do the job. In this situation, the Neural Network might be monstrous overkill.

You'll probably notice that there is a lot of "maybe" and "may" and "it's possible" in my answer above. Unfortunately, there tend to be no hard and fast rules. When I think about what model to use, I think of the following criteria, which at least puts me in the right direction (hopefully):

  1. What are the computational constraints? (For example, if I need to provide fast estimates to a user on a webpage based on dynamic inputs, I won't be using a deep network).

  2. What does the data look like when it's visualized? For regression problems, does the input resemble any mathematical functions I know? Can any transformations be applied to it? For clustering, are there any obvious patterns? How are the clusters shaped, approximately?

  3. How sparse or dense is the data set? What is the propensity for overfitting? How easy would it be to perform cross-validation?

After that, it's a matter of using experience and doing some experimental tests, then improving on it. It's just as much an art as it is a science, which makes it as tough as it is exciting.

I hoped that helped; tell me if you have any questions. Good luck!

EDIT: As a short addendum, and in answer to the specific questions you posed: expressive models such as NN are useful when the data contains many rich relationships; however, the models you call "rigid" can be very useful when complexity is a major issue and/or when the general form of the data can be predicted in some way (for example, "the data looks like a linear function with some noise").

Edit edit: Wait, one more thing - I've never found the approach of "unleash all the algorithms and see what sticks" to be useful. In my opinion, the single most important skills one can cultivate in this field is to understand why certain problems are best solved by certain solutions; it'll save you huge amounts of time.

  • $\begingroup$ I wonder if a decision tree could be produced for deciding which type of model is best, given the answers the the questions you pose? $\endgroup$
    – naught101
    Dec 3, 2013 at 23:47
  • $\begingroup$ It most certainly could - various types of mixture or ensemble models are quite common, and automated model selection is still a new and exciting field (not to mention genetic algorithms and other metaheuristics which can completely construct solutions from scratch). However, for very general cases like the one above, such solutions are so terrifyingly inefficient that they aren't even worth considering when an experienced and knowledgeable person can do some experiments and find a model which is much more useful. There isn't a single silver bullet or magic hammer, at least not yet. $\endgroup$ Dec 4, 2013 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.