I know that we can use training data error to know if the model is underfitting, but it tells us more about the model than the data. How to know if the data is just hard to model? Are there any metrics to assess how hard the data is to model?
$\begingroup$
$\endgroup$
10
-
4$\begingroup$ Sort of related: How to know that your machine learning problem is hopeless? $\endgroup$– Stephan KolassaOct 1 at 18:40
-
23$\begingroup$ Perhaps a stupid question: what is the difference between "the data is too complicated for the model" and "the model is too simple for the data"? $\endgroup$– Daniel WagnerOct 1 at 18:43
-
2$\begingroup$ I think it's the difference between "the world is too complicated" and "the researcher is not smart enough", but how the statistics can tell us which is which is the main point! $\endgroup$– Nick CoxOct 1 at 20:30
-
$\begingroup$ If the correct model is a very complex one, say .... with interactions and splines and such, then the model is too simple. Maybe MARS would work great, but the researcher has never heard of it. OTOH, some data are just really messy. Social science data often is. So is a lot of medical data. We people are complex creatures! $\endgroup$– Peter FlomOct 1 at 20:39
-
6$\begingroup$ @DanielWagner that isn't the distinction in the question... Two examples when using a linear model: An example of "data is hard to model" would be data from the distribution $y = \beta x + N(0,100)$. The correct model is simple, but it fits the data very poorly. An example of "model is too simplistic" would be $y = \beta x + \alpha x^2 + N(0,1)$. The linear model is too simplistic. $\endgroup$– usulOct 2 at 15:07
|
Show 5 more comments
3 Answers
$\begingroup$
$\endgroup$
4
I don't know of any metrics, exactly, but there are a bunch of indications.
Substantive knowledge. What usually happens in your field? Way back when I was in grad school, I was at a lunch for fellowship winners. A psychologist and a physicist were talking and the physicist said "if we get R square of 0.9, we are upset that it's so low" to which the psychologist said "if we get R square that high, we know we did something wrong!"
Graphs. Graph your data in all sorts of ways. Do the relationships fall on nice straight lines? Quadratic curves? Or what?
Measurement error - How reliable are your measurements? Models work better with precisely measured variables. Latent variables can be particularly hard to measure accurately.
Also, how valid are your measurements? Measuring height is pretty straightforward. Measuring (say) strength of a national economy, not so much.
Interactions. Interactions are notorious. And the reliability of an interaction between two variables that are not very reliable is even less reliable.
Clarity and strength of underlying theory. This is related to the first point, but, how clear is your theory? Are you doing highly exploratory work, or are you confirming a well-known model for a particular population?
I am sure there are more such things, but this is a start.
-
1$\begingroup$ Great list. Can you give an example of "the reliability of an interaction between two variables that are not very reliable", in a problem that isn't very badly-defined? $\endgroup$– smciOct 2 at 0:22
-
1$\begingroup$ Poor reliability means lots of noise, and interactions are typically weaker. $\endgroup$ Oct 2 at 1:34
-
$\begingroup$ Sure. How about the interaction of intelligence and any of the "Big 5" personality factors. $\endgroup$ Oct 2 at 8:58
-
$\begingroup$ @PeterFlom: how is "intelligence" precisely defined, how is it measured, is it a single quantity or multiple distinct quantities? (IQ is not intelligence, only a proxy for it, AFAIK). (I had said "problem that isn't very badly-defined") $\endgroup$– smciOct 4 at 19:38
$\begingroup$
$\endgroup$
1
Set aside some of your training data as a hold-out dataset. Fit your model on the rest. As you make your model more complex, does its performance on the hold-out dataset consistently improve? If it does, your model may be under-fitted. If not, it may already be close to optimal, or possibly over-fit.
-
2$\begingroup$ And the paradigm of structural risk minimization formalizes a way to do this rigorously. $\endgroup$– usulOct 2 at 15:15
$\begingroup$
$\endgroup$
Benchmark with models of different size
A common practice for developing machine learning models is to not only look at the best model you have, but also measure and visualize the relationship of model performance (on some validation set) with respect to model size to observe any trends visible. If a slightly smaller models than the largest one show weaker performance, then that suggests that further increases are likely to fit data better; but if your performance has reached a plateau, this indicates that the data might just be fundamentally hard to model.
The same applies to other aspects of model structure complexity - if simpler models reach similar results as complex models, that's an indication that what you get might be a fundamental limitation of data, but if more complex models perform better, then that's an indication that perhaps even the most complex model you tried is "too weak" to properly model the data and going even further could improve things.
Benchmark against expert human performance
Many tasks try to automate something which can also be performed by human judgement. Having a validation dataset processed by expert humans provides a benchmark of what is definitely possible to extract from that data (because you observed a "system" doing that), so a sufficiently powerful model should be able to perform at least as well. If the gap between model performance and human performance is large, that indicates that a more powerful model (or, potentially, more training data or more external world knowledge) might get better results, but if the gap is insignificant then that suggests that perhaps the problem fundamentally can't be modeled better.
But, as you might notice from my wording, there generally is no certainty. There is an active (and important!) niche of research on trying to provide formal guarantees, but (as far as I know) it has not yet resulted in generally applicable techniques to determine these limits for practical problems before actually trying to solve the particular task.
