# Are all models useless? Is any exact model possible — or useful?

This question has been festering in my mind for over a month. The February 2015 issue of Amstat News contains an article by Berkeley Professor Mark van der Laan that scolds people for using inexact models. He states that by using models, statistics is then an art rather than a science. According to him, one can always use "the exact model" and that our failure to do so contributes to a "lack of rigor ... I fear that our representation in data science is becoming marginalized."

I agree that we are in danger of becoming marginalized, but the threat usually comes from those who claim (sounding a lot like Professor van der Laan, it seems) that they are not using some approximate method, but whose methods are in fact far less rigorous than are carefully applied statistical models -- even wrong ones.

I think it is fair to say that Prof van der Laan is rather scornful of those who repeat Box's oft-used quote, "all models are wrong, but some are useful." Basically, as I read it, he says that all models are wrong, and all are useless. Now, who am I to disagree with a Berkeley professor? On the other hand, who is he to so cavalierly dismiss the views of one of the real giants in our field?

In elaborating, Dr van der Laan states that "it is complete nonsense to state that all models are wrong, ... For example, a statistical model that makes no assumptions is always true." He continues: "But often, we can do much better than that: We might know that the data are the result of $n$ independent identical experiments." I do not see how one can know that except in very narrow random-sampling or controlled experimental settings. The author points to his work in targeted maximum likelihood learning and targeted minimum loss-based learning, which "integrates the state of the art in machine learning/data-adaptive estimation, all the incredible advances in causal inference, censored data, efficiency and empirical process theory while still providing formal statistical inference." Sounds great!

There are also some statements I agree with. He says that we need to take our work, our role as a statistician, and our scientific collaborators seriously. Hear hear! It is certainly bad news when people routinely use a logistic regression model, or whatever, without carefully considering whether it is adequate to answering the scientific question or if it fits the data. And I do see plenty of such abuses in questions posted in this forum. But I also see effective and valuable uses of inexact models, even parametric ones. And contrary to what he says, I have seldom been "bored to death by another logistic regression model." Such is my naivety, I guess.

So here are my questions:

1. What useful statistical inferences can be made using a model that makes no assumptions at all?
2. Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?
3. Are all inexact models indeed useless?
4. Is it possible to know that you have the exact model other than in trivial cases?
5. If this is too opinion-based and hence off-topic, where can it be discussed? Because Dr van der Laan's article definitely does need some discussion.
• I'd say a model with no assumptions is almost always useless, but van der Laan is surely making assumptions without admitting it. We don't really know many things for certain so his "We might know that the data are the result of n independent identical experiments" is in fact an assumption, or at least partially so. We might have good reason to suppose they're identical and independent but in practice we see that very often it turns out that it wasn't truly the case (and if we discover that sometimes, it must be undiscovered sometimes, presumably much more often). – Glen_b Apr 2 '15 at 2:25
• A model without assumptions is just a replication of your datamatrix. It is true, but also a useless duplication of effort. One could also argue that such a model is not a model: One definition of a model is that it is a simplification of reality. So a model that does not simplify is by definition not a model. That simplification has a purpose: it helps us substantively understand what is going on; it gives us a main story. If you don't care about understanding why things happen and only want to predict, that main story is irrelavant. However, some want to do more than create a cristal ball... – Maarten Buis Apr 2 '15 at 7:22
• A model without assumptions is not a model. The model IS a collection of assumptions. Also, as @Glen_b says, iid assumption is rather strong, and I can't think of an example where it is "true". Even when repeatedly flipping a coin, there are correlations (see work of Diaconis). I have only skimmed the article but it seems short sighted. – P.Windridge Apr 2 '15 at 9:49
• The author's first example about flying to Mars is bogus. Newtonian mechanics (which is relied on for many aspects of space missions) is a wrong model, but we rely on it and it is useful. – P.Windridge Apr 2 '15 at 10:20
• Yes, Newtonian mechanics is a great example of George Box's point. – Glen_b Apr 2 '15 at 10:49

The cited article seems to be based on fears that statisticians "will not be an intrinsic part of the scientific team, and the scientists will naturally have their doubts about the methods used" and that "collaborators will view us as technicians they can steer to get their scientific results published." My comments on the questions posed by @rvl come from the perspective of a non-statistician biological scientist who has been forced to grapple with increasingly complicated statistical issues as I moved from bench research to translational/clinical research over the past few years. Question 5 is clearly answered by the multiple answers now on this page; I'll go in reverse order from there.

4) It doesn't really matter whether an "exact model" exists, because even if it does I probably won't be able to afford to do the study. Consider this issue in the context of the discussion: Do we really need to include “all relevant predictors?” Even if we can identify "all relevant predictors" there will still be the problem of collecting enough data to provide the degrees of freedom to incorporate them all reliably into the model. That's hard enough in controlled experimental studies, let alone retrospective or population studies. Maybe in some types of "Big Data" that's less of a problem, but it is for me and my colleagues. There will always be the need to "be smart about it," as @Aksakal put it an an answer on that page.

In fairness to Prof. van der Laan, he doesn't use the word "exact" in the cited article, at least in the version presently available on line from the link. He talks about "realistic" models. That's an important distinction.

Then again, Prof. van der Laan complains that "Statistics is now an art, not a science," which is more than a bit unfair on his part. Consider the way he proposes to work with collaborators:

... we need to take the data, our identity as a statistician, and our scientific collaborators seriously. We need to learn as much as possible about how the data were generated. Once we have posed a realistic statistical model, we need to extract from our collaborators what estimand best represents the answer to their scientific question of interest. This is a lot of work. It is difficult. It requires a reasonable understanding of statistical theory. It is a worthy academic enterprise!

The application of these scientific principles to real-world problems would seem to require a good deal of "art," as with work in any scientific enterprise. I've known some very successful scientists, many more who did OK, and some failures. In my experience the difference seems to be in the "art" of pursing scientific goals. The result might be science, but the process is something more.

3) Again, part of the issue is terminological; there's a big difference between an "exact" model and the "realistic" models that Prof. van der Laan seeks. His claim is that many standard statistical models are sufficiently unrealistic to produce "unreliable" results. In particular: "Estimators of an estimand defined in an honest statistical model cannot be sensibly estimated based on parametric models." Those are matters for testing, not opinion.

His own work clearly recognizes that exact models aren't always possible. Consider this manuscript on targeted maximum likelihood estimators (TMLE) in the context of missing outcome variables. It's based on an assumption of outcomes missing at random, which may never be testable in practice: "...we assume there are no unobserved confounders of the relationship between missingness ... and the outcome." This is another example of the difficulty in including "all relevant predictors." A strength of TMLE, however, is that it does seem to help evaluate the "positivity assumption" of adequate support in the data for estimating the target parameter in this context. The goal is to come as close as possible to a realistic model of the data.

2) TMLE has been discussed on Cross Validated previously. I'm not aware of widespread use on real data. Google Scholar showed today 258 citations of what seems to be the initial report, but at first glance none seemed to be on large real-world data sets. The Journal of Statistical Software article on the associated R package only shows 27 Google Scholar citations today. That should not, however, be taken as evidence about the value of TMLE. Its focus on obtaining reliable unbiased estimates of the actual "estimand" of interest, often a problem with plug-in estimates derived from standard statistical models, does seem potentially valuable.

1) The statement: "a statistical model that makes no assumptions is always true" seems to be intended as a straw man, a tautology. The data are the data. I assume that there are laws of the universe that remain consistent from day to day. The TMLE method presumably contains assumptions about convexity in the search space, and as noted above its application in a particular context might require additional assumptions.

Even Prof. van der Laan would agree that some assumptions are necessary. My sense is that he would like to minimize the number of assumptions and to avoid those that are unrealistic. Whether that truly requires giving up on parametric models, as he seems to claim, is the crucial question.

• Very nice answer. The book Targeted Learning is a good place to learn more. In addition to theory it contains some case studies. – NRH May 5 '16 at 22:35

Maybe I missed the point, but I think you have to step back a little bit.

I think his point is the abuse of easy-accessible tools with no further knowledge. This is also true for a simple t-test: just feed the algorithm with your data, getting a p<0.05 and thinking, that your thesis is true. Completely wrong. You, of course, have to know more about your data.

Stepping even further back: There is nothing like an exact model (physicist here). But some agree very well with our measurements. The only exact thing is math. Which has nothing to do with reality or models of it. Everything else (and every model of the reality) is "wrong" (as quoted so often).

But what does mean "wrong" and useful? Judge by yourself:

ALL of our current high-tech (computers, rockets, radioactivity etc) is based on these wrong models. Maybe even computed by "wrong" simulations with "wrong" models.
-> Focus more on the "useful" instead of "wrong";)

1. Don't know, sorry!
2. Yes. One example: in particle-physics, you want to detect certain particles (say electrons, protons etc.). Every particle leaves a characteristic trace in the detector (and therefore the data), but varies even for the same particle (by its nature). Today, most of the people use machine-learning to achieve this goal (this was a huge simplification, but it is pretty much like this) and there is an increase in efficiency of 20%-50% compared to doing it by hand statistics.
3. Nobody really claimed this! Don't make wrong conclusion! (a: all models are inexact and b: some are useful. Don't confuse things)
4. There is no thing as an exact model (except in math, but not really in statistics as having points exactly on a straight line and "fitting" a line through it may be exact... but that's an uninteresting special case which never happens).
5. Don't know :) But IMHO I see this more as a "just because every child can use it, not everyone should" and don't overuse it blindly.
• I'll consider your points, but on (3) and (4), I suggest you look at van der Laan's article, as well as my question, because the article does say pretty clearly that he has no use for inexact models, and repeatedly refers to "the exact model". So yes, I'd say that somebody indeed said that. I pretty much agree with what you say about models, as well as mechanical application of methods; and so I don't think it's me you want to be asking to step back. – Russ Lenth Apr 27 '16 at 21:40
• Yes, to all. And mostly, I meant him to step back, for sure ;) I think, what I did not mention, to build theoretical (mathematical) models, you do of course need "exact" models. So to "invent" better models and statistical tools, you need deterministic (or exact) models as a base. Even dough targeted learning does not have to be "inexact" in this sense, I think. – Mayou36 Apr 28 '16 at 12:08

In econ, much is said of understanding the 'data generating process.' I'm not sure what exactly is meant by an 'exact' model, but in econ it might be the same as a 'correctly specified' model.

Certainly, you want to know as much about the process that generated the data as you can before attempting a model, right? I think the difficulty comes from a) we may not have a clue about the real DGP and b) even if we knew the real DGP it might be intractable to model and estimate (for many reasons.)

So you make assumptions to simplify matters and reduce estimation requirements. Can you ever know if your assumptions are exactly right? You can gain evidence in favor of them, but IMO it's tough to be really sure in some cases.

I have to filter all of this in terms of both established theory as well as practicality. If you make an assumption consistent with a theory and that assumption buys you better estimation performance (efficiency, accuracy, consistency, whatever) then I see no reason to avoid it, even if it makes the model 'inexact'.

Frankly, I think the article is meant to stimulate those who work with data to think harder about the entire modeling process. It's clear that van der Laan makes assumptions in his work. In this example, in fact, van der Laan seems to throw away any concern for an exact model, and instead uses a mish-mash of procedures to maximize performance. This makes me more confident that he raised Box's quote with the intent of preventing people from using it as an escape from the difficult work of understanding the problem.

Let's face it, the world is rife with misuse and abuse of statistical models. People blindly apply whatever they know how to do, and worse, others often interpret the results in the most desirable way. This article is a good reminder to be careful, but I don't think we should take it to the extreme.

The implications of the above for your questions:

1. I agree with others on this post that have defined a model as a set of assumptions. With that definition, a model with no assumptions isn't really a model. Even exploratory data analysis (i.e. model free) requires assumptions. For example, most people assume the data are measured correctly.
2. I don't know about TMLE, per se, but in economics there are many articles that use the same underlying philosophy of inferring about a causal effect on an unobserved counterfactual sample. In those cases, however, receiving a treatment is not independent of the other variables in the model (unlike TMLE), and so economists make extensive use of modeling. There are a few case studies for structural models, such as this one where the authors convinced a company to implement their model and found good results.
3. I think all models are inexact, but again, this term is a bit fuzzy. IMO, this is at the core of Box's quote. I'll restate my understanding of Box this way: 'no model can capture the exact essence of reality, but some models do capture a variable of interest, so in that sense you might have a use for them.'
4. I addressed this above. In short, I don't think so.
5. I'm not sure. I like it right here.

To address point 3, the answer, obviously, is no. Just about every human enterprise is based on a simplified model at some point: cooking, building, interpersonal relationships all involve humans acting on some kind of data + assumptions. No one has ever constructed a model that they did not intend to make use of. To assert otherwise is idle pedantry.

It is much more interesting and enlightening, and useful to ask when inexact models are not useful, why they fail in their usefulness, and what happens when we rely on models that turn out not to be useful. Any researcher, whether in academia or industry, has to ask that question shrewdly and often.

I don't think the question can be answered in general, but the principles of error propagation will inform the answer. Inexact models break down when the behavior they predict fails to reflect behavior in the real world. Understanding how errors propagate through a system can help one understand how much precision is necessary in modeling the system.

For example, a rigid sphere is not usually a bad model for a baseball. But when you are designing catcher's mitt, this model will fail you and lead you to design the wrong thing. Your simplifying assumptions about baseball physics propagate through your baseball-mitt system, and lead you to draw the wrong conclusions.

1) What useful statistical inferences can be made using a model that makes no assumptions at all?

A model is by definition a generalization of what you are observing that can be captured by certain causal factors that in turn can explain and estimate the event you are observing. Given that all those generalization algorithms have some sort of underlying assumptions. I am not sure what is left of a model if you have no assumptions whatsoever. I think you are left with the original data and no model.

2) Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?

I don't know. Maximum likelihood is used all the time. Logit models are based on those as well as many other models. They don't differ a whole lot to standard OLS where you focus on the reductions of the sum of the square of the residuals. I am not sure what targeted maximum likelihood is. And, how it differs from traditional maximum likelihood.

3) Are all inexact models indeed useless?

Absolutely not. Inexact models can be very useful. First, they contribute to better understanding or explaining a phenomenon. That should count for something. Second, they may provide descent estimation and forecasting with relevant Confidence Interval to capture the uncertainty surrounding an estimate. That can provide a lot of info on what you are studying.

The issue of "inexact" also raises the issue of the tension between parsimony and overfit. You can have a simple model with 5 variables that is "inexact" but does a pretty good job of capturing and explaining the overall trend of the dependent variable. You can have a more complex model with 10 variables that is "more exact" than the first one (higher Adjusted R Square, lower Standard Error, etc.). Yet, this second more complex model may really crash when you test it using a Hold Out sample. And, in such case maybe the "inexact" model actually performs a lot better in the Hold Out sample. This happens literally all the time in econometrics and I suspect in many other social sciences. Beware of "exact" models. They can often be synonimous with overfit models and mis-specified models (models with non stationary variables that have underlying trends (unit root) with no economic meaning imparted to the model).

4) Is it possible to know that you have the exact model other than in trivial cases?

It is not possible to know that you have the exact model. But, it is possible to know you have a pretty good model. The information criteria measures (AIC, BIC, SIC) can give you much information allowing to compare and benchmark the relative performance of various models. Also, the LINK test can also help in that regard.

5) If this is too opinion-based and hence off-topic, where can it be discussed? Because Dr van der Laan's article definitely does need some discussion.

I would think this is as appropriate a forum to discuss this issue as anywhere else. This is a pretty interesting issue for most of us.

Said article appears to me to be a honest but political article, a sincere polemic. As such, it contains a lot of passionate passages that are scientific non-sense, but that nevertheless may be effective in stirring up useful conversations and deliberations on important matters.

There are many good answers here so let me just quote a few lines from the article to just show that Prof. Laan is certainly not using any kind of "exact model" in his work (and by the way, who says that the "exact model" is a concept equivalent to the actual data generating mechanism?)

Quotes (bold my emphasis)

"Once we have posed a realistic statistical model, we need to extract from our collaborators what estimand best represents the answer to their scientific question of interest."

Comment: "realistic" is as removed from "exact" as the Mars is from the Earth. They both orbit the Sun though, so for some purposes it doesn't matter which planet one chooses. For other purposes, it does matter. Also "best" is a relative concept. "Exact" is not.

"Estimators of an estimand defined in an honest statistical model cannot be sensibly estimated based on parametric models...

Comment: Honesty is the best policy indeed, but it is certainly not guaranteed to be "exact". Also, "sensible estimation" appears to be a very diluted outcome if one uses the "exact model".

"In response to having to solve these hard estimation problems the best we can, we developed a general statistical approach..."

Comment: OK. We are "doing the best we can". As almost everybody is thinking about oneself. But "best we can" is not "exact".

This post was brought to my attention just a few days ago. Thank you for your interest.

### Question 1: What useful statistical inferences can be made using a model that makes no assumptions at all?

Before I answer this question we should agree on a definition of the word model:

A common definition of a statistical model is the set of possible probability distributions or densities of the observed data. In addition to formulating a statistical model, one might make additional assumptions that do not restrict the distribution of the data such as missing at random, coarsening at random, or randomization assumptions in so called censored or missing data models. These latter type of assumptions are typically non-testable, i.e. they do not put restrictions on the distribution of the data and can thus not be tested based on data. For example, one commonly represents the observed data as a missing or censored data structure on a full-data random variable, and defines the target quantity of interest as some feature of the full-data distribution. To establish identification of this target quantity of the full-data distribution, one needs to make certain assumptions such as the ones I mention above. These assumptions allow us to define an estimand (i.e. feature of the distribution of the observed data) that equals the desired target quantity, even though these assumptions do not put any restrictions on the distribution of the data. These non-testable assumptions do not affect the statistical estimation or statistical properties of estimators of the target estimand, but they do affect the interpretation of the target estimand and the degree one feels comfortable extending the purely statistical interpretation to a causal or full-data distribution interpretation.

I am going to focus on the notion of a statistical model. If we make no assumptions at all, then the statistical model would be all possible probability distributions. I agree that in this case we cannot do anything. This could happen, but still might be a useful realization, making us careful to over-interpret results that will be derived from statistical models that make assumptions. For example, if one observes a single microarray of gene expressions, then one might have to acknowledge that there is no basis for statistical inference without making very strong, unrealistic assumptions.

In many studies we know or feel highly confident (based on understanding of the experiment) that the data set is the result of independent and identical experiments, in which case we view our data as n independent and identically distributed random variables with a common probability distribution. In other cases, one might condition on the units and treat the data observed on these units as the result of independent experiments, one for each unit. This does not only apply to experiments that involve random sampling of units from a population. For example, a study that enrolls patients that satisfy some eligibility criterion and then tracks patients longitudinally over time could be thought of as independent experiments, maybe identical, or maybe only independent. Our statistical model may then assume nothing else. Still, this is a real statistical model that allows us to formulate estimators with statistical inference based on asymptotic linearity of estimators and central limit theorems (that work under only assuming independence, or even weaker forms of independence assumptions). We might be able to make more assumptions, such as the treatment variable given the set of observed pre-treatment covariates only depends on a certain subset of the covariates (something that one might learn from talking to the people who made the treatment decision).

In our research we carry out a lot of work on sample size one problems such as observing a single time series over many time points (assuming some form of stationarity), a single community of individuals connected through a network (assuming that the data at next time point on a subject is conditionally independent of data collected on other subjects at that time point, given the data we have observed on the friends of that subject), or a sequential adaptive trial in which the next experiment (e.g. randomly sampling a next group of subjects) is set in response to what is observed in the previous experiments. Again, such types of studies satisfy conditional independence assumptions that allow for estimators with asymptotic statistical inference. For me, a realistic statistical model is a model that is known to contain the true probability distribution of the data, or at least it can be sensibly defended as a truthful statistical model.

One should be ready to defend a statistical model. Note that, using your language, an "exact model" would be a model that contains the true probability distribution of the data, but has nothing to do with making a lot of assumptions. The only hope to succeed in formulating such a model is to work hard on understanding the data generating experiment, learning about independence and conditional independence assumptions. There are cases where one cannot be sure (e.g. models for a single time series that avoid making parametric assumptions but still need a form of stationarity), even when posing a highly nonparametric model, and, the assumed model might be going as far as possible while still being able to obtain statistical inference (based on state of the art advances in probability theory). Even then (heavily advancing on current statistical methods), it is fair and necessary to criticize and be fully aware of the assumptions, while still moving forward with valid statistical estimators for such a model. This still represents important advances relative to working with parametric models that are known to be false from the start and cannot be defended at all. It might motivate us to more carefully design experiments for which this same model will be known to be valid, where we now know that we actually have valid powerful methods that handle such highly challenging statistical models.

The selection of a statistical model should be distinguished from the construction of an estimator that might try out many working models and machine learning algorithms as a way to approximate the true distribution of the data. The fit of a data distribution is not a model, but just the realization of an estimator of the data distribution.

Another important benefit by having defined the statistical estimation problem realistically is that one can set up simulation studies to evaluate the behavior of estimators (and data set competitions), refine them, learn the weak spots, and propose a bootstrap respecting the true experiment to further improve on finite sample inference. In the end, the asymptotic results are a must, but all that matters is finite sample inference, so one should always aim to work on finite sample improvements without affecting asymptotic optimality. Even such finite sample improvements are often guided by theory.

### Question 2: Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?

There is a growing literature on Targeted Learning. TMLE started with a 2006 article, and we published two books on the topic (van der Laan, Rose, 2011 and 2018) including contributions from a variety of authors working in the area. I just found out that the 2018 book (Targeted Learning in Data Science) is the top 1 in the Springer Series of Statistics over last three years, while the 2011 book is in top 3% overall going back to beginning of this series. Similarly, we see a great demand for workshops on the topic which we are giving regularly. We recently gave a workshop at the Bill and Melinda Gates Foundation on Targeted Learning and it included an initial presentation which showcased case studies in journals such as the New England Journal of Medicine, among others. There will be a link posted since it was recorded, feel free to contact me about it. Of course, these papers can all be Googled but this may still be helpful. Overall, I now regularly encounter articles by authors I do not know (e.g. not former students, postdocs, etc.). This is a good thing, and it is a joy to see new Ph.D. students at other places contributing new insights. Sometimes it is painful to see how some of such contributions are simply not understanding the material and are confusing the literature. Still, many of such authors are making a concerted effort, so they will get there eventually.

### Question 3: Are all inexact models indeed useless?

If we define "inexact models" as statistical models that can be defended but for which we have no guarantee that the true data distribution is in it, then in my answer to Question 1 I clarify that such models are still useful. Work in such models advance the literature: the assumptions are transparent and for anybody to criticize and evaluate; and once one realizes the kind of assumptions ond needs to worry about, these are realistic enough so that one can expect future applications in which they can be applied. A model where not a single person on earth believes them is not helpful at all.

For example, we teach our students that in GEE using a parametric regression model for a multivariate outcome that, for each choice of covariance matrix of the residuals, the estimator of the coefficients is consistent and asymptotically linear; and if we estimate the covariance matrix consistently then the estimator is efficient. These statements are predicated on the regression model being correct, but since they are not, they actually teach the wrong thing. In the real world, 1) the choice of covariance matrix defines the projection of the true regression curve on the parametric working model, and thereby affects the target estimand (so confidence intervals for two different covariance matrices will be non-overlapping for large enough sample size); 2) the variability of the estimator of the covariance matrix heavily contributes to the variability of the estimator of the coefficients, so that more nonparametric (and thus consistently) estimation of the covariance matrix typically heavily increases the actual variance of the estimator. The majority that one is taught in statistics is based on such unrealistic assumptions and is actually wrong when applied to the real world. This is just one of the million examples in which what we teach based on these models is not even representative of what happens in the real world when applying these methods.

### Question 4: Is it possible to know that you have the exact model other than in trivial cases?

Not at all, as I explain in my response to Question 1. Either way, it is clearly my philosophy that one should make a sincere effort to define the real statistical estimation problem as accurately as possible, making assumptions that are reasonable and one can also defend. To me, this honest formulation is better than posing models in which the whole world knows the assumptions are plain wrong -- the confidence intervals have asymptotic coverage zero and the p-values result in testing procedures that have asymptotic type I error equal to 1. In addition, when people use these wrong models they typically play with them and try out many (I cannot blame them when that is only tool one has available), resulting in additional bias beyond the issue of using a statistical method with asymptotic coverage zero (for the assumed question of interest) and type I error 1.

### You wrote: "If this is too opinion-based and hence off-topic, where can it be discussed? Because Dr van der Laan's article definitely does need some discussion."

This gets to the essence of statistical learning. Yes, this is incredibly important and it changes the way one approaches statistics. We often refer to the following steps as the roadmap of statistical learning (the answer to a statistical query): 1) Define data; 2) Define probability distribution of data and our knowledge about the data generating experiment; 3) Define target estimand (possibly augmenting its statistical interpretation with a causal/enhanced interpretation under specified non-testable assumptions); 4) Define estimator that is asymptotically valid under the statistical model assumptions; 5) Obtain inference based on sampling distribution of estimator; 6) Interpret the results (e.g. augmenting with sensitivity analysis to allow for interpretation of target estimand going in between purely statistical and purely causal).

By taking these steps seriously, one often ends up with new statistical estimation problems. That itself can be an important contribution. In addition, many times new identification results (i.e, causal inference), new estimators and new theory needs to be developed (e.g. statistical estimators developed within the TMLE template), but this only happens because one has defined the precise challenge so that expertise and brainpower can be brought in by the general scientific community to solve it. If we replace the real problem by a toy problem, we avoid the real challenges.

• Thank you for your response. It is going to take me a while to digest it. But I wonder to what extent your thinking is restricted to massive datasets. Take, for example, Yates’s experiment on oat yields - a classic split-plot experiment with 72 observations total. If I were to ask you to assess how the three varieties differ, or the effect of nitrogen, how would you approach it that would differ from what Yates did? I know this is a old example, but there are still people doing experiments not unlike this one, in agriculture, social science, ... – Russ Lenth Nov 27 '19 at 2:22
• On behalf of this site, Mark, I want to express my gratitude for the interest you have shown and the effort you put into creating this thoughtful post. – whuber Dec 4 '19 at 20:08

(I don't see the phrase "exact model" in the article (though quoted above))

1) What useful statistical inferences can be made using a model that makes no assumptions at all?

You have to start somewhere. If that's all you have (nothing), it can be a starting point.

2) Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?

To answer the second question, Targeted Maximum Likelihood turns up in 93/1143281 (~.008% ) of papers in arxiv.org. So, no is probably a good estimate (without assumptions) to that one.

3) Are all inexact models indeed useless?

No. Sometimes you only care about one aspect of a model. That aspect can be very good and the rest very inexact.

4) Is it possible to know that you have the exact model other than in trivial cases?

The best model is the model that best answers your question. That may mean leaving something out. What you want to avoid, as best you can, is assumption violation.

5) Happy hour. And drinks are cheaper to boot!

I find use of the word "exact" a bit unsettling. It's not very statistician-like talk. Inexactitude? Variation? Thank G-d! That's why we are all here. I think the phrase "All models are wrong..." is okay, but only in the right company. Statisticians understand what it means, but few others do.

• Good point about the phrase "exact model." He says things about "true" and "actual" models that are equivalent, especially given the context and tone of the article, but you're right, the quote is not "exact" (pun intended). – SQLServerSteve May 5 '16 at 1:03
• My bad. I should have quoted him correctly. – Russ Lenth May 5 '16 at 1:19
• I feel that trying to achieve the true or actual model misses the point of the exercise. I think what he's really is trying to discuss is bad models. – mandata May 5 '16 at 1:27
• Yeah the article makes many good points, but he intersperses a lot of extreme comments like "It is complete nonsense to state that all models are wrong," when they are in fact all wrong to some degree. It is nonsense to say otherwise. He's definitely staking an extreme position. Rvl was entirely right to bring this up and call him out (don't sweat the quote Ryl, the important thing is that you captured the meaning correctly). – SQLServerSteve May 5 '16 at 2:13

I'm going to approach this from the alternate direction of philosophy, in light of the really useful principles of Uncertainty Management discussed in George F. Klir's books on fuzzy sets. I can't give van der Laan exactness, but I can provide a somewhat exhaustive case for why his goal is logically impossible; that is going to call for a lengthy discussion that references other fields, so bear with me.

Klir and his co-authors divide uncertainty into several subtypes, such as nonspecificity (i.e. when you have an unknown set of alternatives, dealt with through means like the Hartley Function); imprecision in definitions (i.e. the "fuzziness" modeled and quantified in fuzzy sets); strife or discord in evidence (addressed in Dempster-Shafer Evidence Theory); plus probability theory, possibility theory and measurement uncertainty, where the goal is to have an adequate scope to capture the relevant evidence, while minimizing errors. I look at the whole toolbox of statistical techniques as alternate means of partitioning uncertainty in different ways, much like a cookie cutter; confidence intervals and p-values quarantine uncertainty in one way, while measures like Shannon's Entropy whittle it down from another angle. What they can't do, however, is eliminate it entirely. To achieve an "exact model" of the kind van der Laan seems to describe, we'd need to reduce all of these types of uncertainty down to zero, so that there's no more left to partition. A truly "exact" model would always have probability and possibility values of 1, nonspecificity scores of 0 and no uncertainty whatsoever in the definitions of terms, ranges of values or measurement scales. There would be no discord in alternate sources of evidence. The predictions made by such a model would always be 100 percent accurate; predictive models essentially partition their uncertainty off into the future, but there would be none left to put off. The uncertainty perspective has some important implications:

• This tall order is not only physically implausible, but actually logically impossible. Obviously, we cannot achieve perfectly continuous measurement scales with infinitesimal degrees, by gathering finite observations using fallible, physical scientific equipment; there will always be some uncertainty in terms of measurement scale. Likewise, there will always be some fuzziness surrounding the very definitions we employ in our experiments. The future is also inherently uncertain, so the supposedly perfect predictions of our "exact" models will have to be treated as imperfect until proven otherwise - which would take an eternity.

• To make matters worse, no measurement technique is 100 percent free of error at some point in the process, nor can it be made comprehensive enough to embrace all of the possibly conflicting information in the universe. Furthermore, the elimination of possible confounding variables and complete conditional independence cannot be proven thoroughly without examining all other physical processes that affect the one we're examining, as well as those that affect these secondary processes and so on.

• Exactness is possible only in pure logic and its subset, mathematics, precisely because abstractions are divorced from real-world concerns like these sources of uncertainty. For example, by pure deductive logic, we can prove that 2 + 2 = 4 and any other answer is 100 percent incorrect. We can also make perfectly accurate predictions that it will always equal 4. This kind of precision is only possible in statistics when we're dealing with abstractions. Statistics is incredibly useful when applied to the real world, but the very thing that makes it useful injects at least some degree of inescapable uncertainty, thereby rendering it inexact. It is an unavoidable dilemma.

• Furthermore, Peter Chu raises additional limitations in the comments section of the article rvl linked to. He puts it better than I can:

"This solution surface of NP-hard problems is typically rife with many local optima and in most case it is computationally unfeasible to solve the problem i.e. finding the global optimal solution in general. Hence, each modeler is using some (heuristic) modeling techniques, at best, to find adequate local optimal solutions in the vast solution space of this complex objective function."

• All of this means that science itself cannot be perfectly accurate, although van der Laan seems to speak of it in this way in his article; the scientific method as an abstract process is precisely definable, but the impossibility of universal and perfect exact measurement means it cannot produce exact models devoid of uncertainty. Science is a great tool, but it has limits.

• It gets worse from there: Even if were possible to exactly measure all of the forces acting on every constituent quark and gluon in the universe, some uncertainties would still remain. First, any predictions made by such a complete model would still be uncertain due to the existence of multiple solutions for quintic equations and higher polynomials. Secondly, we cannot be completely certain that the extreme skepticism in embodied in the classic question "maybe this is all a dream or a hallucination" is not a reflection of reality - in which case all of our models are indeed wrong in the worst possible way. This is basically equivalent to a more extreme ontological interpretation of the original epistemological formulations of philosophies like phenomenalism, idealism and solipsism.

• In his 1909 classic Orthodoxy G.K. Chesterton noted that the extreme versions of these philosophies can indeed be judged, but by whether or not they drive their believers into mental institutions; ontological solipsism, for example, is actually a marker of schizophrenia, as are some of its cousins. The best that we can achieve in this world is to eliminate reasonable doubt; unreasonable doubt of this unsettling kind cannot be rigorously done away with, even in a hypothetical world of exact models, exhaustive and error-free measurements. If van der Laan aims at ridding us of unreasonable doubt then he is playing with fire. By grasping at perfection, the finite good we can do will slip through our fingers; we are finite creatures existing in an infinite world, which means the kind of complete and utterly certain knowledge van der Laan argues for is permanently beyond our grasp. The only way we can reach that kind of certainty is by retreating from that world into the narrower confines of the perfectly abstract one we call "pure mathematics." This does not mean, however, that a retreat into pure mathematics is the solution to eliminating uncertainty. This was essentially the approach taken by the successors of Ludwig Wittgenstein (1889-1951), who drained his philosophy of logical positivism of whatever common sense it had by rejecting metaphysics altogether and retreating entirely into pure math and scientism, as well as extreme skepticism, overspecialization and overemphasis on exactness over usefulness. In the process, they destroyed the discipline of philosophy by dissolving it into a morass of nitpicking over definitions and navel-gazing, thereby making it irrelevant to the rest of academia. This essentially killed the whole discipline, which had still been at the forefront of academic debate until the early 20th Century, to the point where it still garnered media attention and some of its leaders were household names. They grasped at a perfect, polished explanation of the world and it slipped through their fingers - just as it did through the mental patients GKC spoke of. It will also slip out of the grasp of van der Laan, who has already disproved his own point, as discussed below. The pursuit of models that are too exact is not just impossible; it can be dangerous, if taken to the point of self-defeating obsession. The pursuit of that kind of purity rarely ends well; it's often as self-defeating as those germophobes who scrub their hands so furiously that they end up with wounds that get infected. It's reminiscent of Icarus trying to steal fire from the Sun: as finite beings, we can have only a finite understanding of things. As Chesterton also says in Orthodoxy, "It is the logician who seeks to get the heavens into his head. And it is his head that splits."

In the light of the above, let me tackle some of the specific questions listed by rvl:

1) A model with no assumptions whatsoever is either a) not aware of its own assumptions or b) must be cleanly divorced from considerations that introduce uncertainty, such as measurement errors, accounting for every single possible confounding variable, perfectly continuous measurement scales and the like.

2) I'm still a newbie when it comes to maximum likelihood estimation (MLE), so I can't comment on the mechanics of target likelihood, except to point out the obvious: likelihood is just that, a likelihood, not a certainty. To derive an exact model requires complete elimination of uncertainty, which probabilistic logic can rarely do, if ever.

3) Of course not. Since all models retain some uncertainty and are thus inexact (except in cases of pure mathematics, divorced from real-world physical measurements), the human race would not have been able to make any technological progress to date - or indeed, any other progress at all. If inexact models were always useless, we'd be having this conversation in a cave, instead of on this incredible feat of technology called the Internet, all of which was made possible through inexact modeling.

Ironically, van der Laan's own model is a primary example of inexactness. His own article sketches out a model of sorts of how the field of statistics ought to be managed, with an aim towards exact models; there are no numbers attached to this "model" yet, no measurement of just how inexact or useless most models are now in his view, no quantification of how far we are away from his vision, but I suppose one could devise tests for those things. As it stands, however, his model is inexact. If it is not useful, it means his point is wrong; if it is useful, it defeats his main point that inexact models aren't useful. Either way, he disproves his own argument.

4) Probably not, because we cannot have complete information to test our model with, for the same reasons that we can't derive an exact model in the first place. An exact model would by definition require perfect predictability, but even if the first 100 tests turn out 100 percent accurate, the 101st might not. Then there's the whole issue of infinitesimal measurement scales. After that, we get into all of the other sources of uncertainty, which will contaminate any Ivory Tower evaluation of our Ivory Tower model.

5) To address the issue, I had to put it in the wider context of much larger philosophical issues that are often controversial, so I don't think it's possible discuss this without getting into opinions (note how that in and of itself is another source of uncertainty) but you're right, this article deserves a reply. A lot of what he says on other topics is on the right track, such as the need to make statistics relevant to Big Data, but there is some impractical extremism mixed in there that should be corrected.

• Who are these "successors of Wittgenstein" who "destroyed the discipline of philosophy"!? The trend of post-war analytic philosophy - think of the later Wittgenstein, Austin, Quine, Kenny, Goodman, Lewis, Davidson, Rorty - seems to be rejection of the tenets of logical positivism, rehabilitation of metaphysics, & a retreat from scientism. (Rorty's comments on Nietzsche & Loyola suggest he might have agreed with Chesterton's point.) As to why Kim Kardashian's a household name rather than Saul Kripke, I suspect other trends have been at work since the early 20th Century. – Scortchi - Reinstate Monica May 6 '16 at 9:28
• These post-war reformers are unknown outside their field precisely because they came after the discipline had receded in the '20s/'30s (after a centuries-long decline the extreme logical positivists merely accelerated) into an irrelevance it has never recovered from. The damage had been already been done. By the '50s other academic disciplines were no longer looking to philosophy for leadership and have since often treated it with outright scorn, for out-of-touch, nitpicking pedantry.. I wish that were not true. The image may not reflect the reality of philosophy, but the taint still remains. – SQLServerSteve May 6 '16 at 19:05
• I'd be happy to discuss this in chat next week if you have time, I don't want to get off on a tangent in this thread, but I suspect our positions aren't that far apart. I think you're entirely right, the movement that led to Kardashian etc. has been under way since the early 20th Century - precisely the time that the discipline of philosophy fell into eclipse (whether it deserved to is another matter; I consider it highly practical and wish it were still highly regarded). – SQLServerSteve May 6 '16 at 19:08
• Comments are not for extended discussion; this conversation has been moved to chat. – Scortchi - Reinstate Monica May 8 '16 at 11:46