I am using a boosted trees classifier which is giving better accuracy then all other linear classifier I tried. I have almost an unlimited training data at my disposal , I wanted to know if there is a saturation point in training beyond which even if you increase the training size there would be no increase in accuracy in the case of boosted trees classifier?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
3
There is a saturation point.
Increasing the size of your training set can't help you surpass the assumptions of your modeling method. For example, if you use a linear model to classify data that is separable in a nonlinear way, you will never get perfect accuracy. As we almost never know the underlying process to its full extent, model mismatch is the norm. As George Box famously said "All models are wrong, but some are useful".
Powerful learning methods like neural networks (aka deep learning) or random forests can push the boundaries a little more than less flexible approaches (e.g. kernel methods), but even for them there is only so much that can be learned. Additionally, the amount of data and other resources you would need to gain worthwhile improvements become excessive at some point.
answered Nov 13, 2015 at 8:08
-
$\begingroup$ Right. One way to think about saturation points is that the accuracy responds like a sigmoid or S-shaped curve which increases monotonically towards some theoretical limit or suprema as the sample size goes to infinity. $\endgroup$ Commented Nov 13, 2015 at 10:10
-
$\begingroup$ Neural networks and deep learning are not the same thing. I can have very useful shallow neural networks. deep learning is generally a subset of neural networks. But theoretically we could have other base models that learn internal abstractions as they go, and they too would be deep learning. $\endgroup$ Commented Nov 13, 2015 at 13:16
-
$\begingroup$ Deep learning and neural networks are currently entirely the same thing. It's just a rebranding. Sure, if you change the base models you could still call it deep learning, but by the time that happens someone will figure out another new name. $\endgroup$ Commented Nov 13, 2015 at 13:27
$\begingroup$
$\endgroup$
Your training dataset needs to be representative of the dataset you'll need to classify. Even if it's huge, if it doesn't capture the corner cases, they'll be misclassified. However, on the other hand, you'll need to be careful of overfitting, if it applies to your case.
Also, if you have a virtually unlimited annotated dataset at your disposal, you can repeatedly and randomly split it in training/validation/testing to make sure you have the best model possible. It will probably take days to run, but I think it will be worth it.
$\begingroup$
$\endgroup$
The key issue in my opinion is that we will never know the underlying process exactly.
- We don't know which factors influence class membership. (I am a firm believer in so-called "tapering effect sizes": essentially, everything has an impact on everything else, just to a smaller and smaller extent.)
- Often enough, we even have problems operationalizing those influencers we do know about. For instance, I'm sure that intelligence influences earnings, but I'm just as sure that "intelligence" is not perfectly (!) measured by IQ tests. Psychologists worry a lot about so-called "construct validity", and rightly so.
- Even if we know a factor and have operationalized it well, we don't know whether its influence is linear, logarithmic or some other weird shape... and we have an entire tag devoted to the problem that a predictor's influence can change over its domain of definition. And I only have logistical regression in my mind as I write this - the same problem will also apply to any other kind of classifier.
- And finally, all these problems are magnified indefinitely by the possibilities for interactions: two-way, three-way, four-way, ...
We might think that collecting more and more data and using more and more sophisticated algorithms will solve these problems. However, the number of "reasonable" models we can fit to any given size of dataset will always grow at least as quickly as the dataset, since there are just so many possible predictors, from the phase of the moon to what your participants ate for breakfast. In the end, you will always be tripped up by the bias-variance tradeoff.
answered Nov 13, 2015 at 14:40
$\begingroup$
$\endgroup$
The maximal performance of the set of possible prediction models has an upper bound. As an example, look at a binary outcome $y$. For simplicitly assume we know that $y = 1$ with prior probability 0.5. This means both outcomes are equally likely. Let $x$ be a vector containing the values for your predictors. By Bayes:
$P(y=1|x)=\frac{P(x|y=1)}{P(x|y=1)+P(x|y=0) }$.
The theoretical best prediction model will predict the $y$ that has a higher likelihood of producing $x$. But unless one of the two terms in the denominator is zero, the theorem of Bayes gives you a non-zero probability of the best prediction being wrong.
The easiest example would be $y$ and $x$ being completely unrelated. Then you predict anything for $y$ and will be always wrong with probability 0.5. And no method can improve on that.
In the best way your algorithm will converge towards the theoretical optimum. Then you will usually not achieve the optimum performance with any finite sample size, but the improvements get smaller and smaller.
