43
$\begingroup$

This question has been festering in my mind for over a month. The February 2015 issue of Amstat News contains an article by Berkeley Professor Mark van der Laan that scolds people for using inexact models. He states that by using models, statistics is then an art rather than a science. According to him, one can always use "the exact model" and that our failure to do so contributes to a "lack of rigor ... I fear that our representation in data science is becoming marginalized."

I agree that we are in danger of becoming marginalized, but the threat usually comes from those who claim (sounding a lot like Professor van der Laan, it seems) that they are not using some approximate method, but whose methods are in fact far less rigorous than are carefully applied statistical models -- even wrong ones.

I think it is fair to say that Prof van der Laan is rather scornful of those who repeat Box's oft-used quote, "all models are wrong, but some are useful." Basically, as I read it, he says that all models are wrong, and all are useless. Now, who am I to disagree with a Berkeley professor? On the other hand, who is he to so cavalierly dismiss the views of one of the real giants in our field?

In elaborating, Dr van der Laan states that "it is complete nonsense to state that all models are wrong, ... For example, a statistical model that makes no assumptions is always true." He continues: "But often, we can do much better than that: We might know that the data are the result of $n$ independent identical experiments." I do not see how one can know that except in very narrow random-sampling or controlled experimental settings. The author points to his work in targeted maximum likelihood learning and targeted minimum loss-based learning, which "integrates the state of the art in machine learning/data-adaptive estimation, all the incredible advances in causal inference, censored data, efficiency and empirical process theory while still providing formal statistical inference." Sounds great!

There are also some statements I agree with. He says that we need to take our work, our role as a statistician, and our scientific collaborators seriously. Hear hear! It is certainly bad news when people routinely use a logistic regression model, or whatever, without carefully considering whether it is adequate to answering the scientific question or if it fits the data. And I do see plenty of such abuses in questions posted in this forum. But I also see effective and valuable uses of inexact models, even parametric ones. And contrary to what he says, I have seldom been "bored to death by another logistic regression model." Such is my naivety, I guess.

So here are my questions:

  1. What useful statistical inferences can be made using a model that makes no assumptions at all?
  2. Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?
  3. Are all inexact models indeed useless?
  4. Is it possible to know that you have the exact model other than in trivial cases?
  5. If this is too opinion-based and hence off-topic, where can it be discussed? Because Dr van der Laan's article definitely does need some discussion.
$\endgroup$
  • 12
    $\begingroup$ I'd say a model with no assumptions is almost always useless, but van der Laan is surely making assumptions without admitting it. We don't really know many things for certain so his "We might know that the data are the result of n independent identical experiments" is in fact an assumption, or at least partially so. We might have good reason to suppose they're identical and independent but in practice we see that very often it turns out that it wasn't truly the case (and if we discover that sometimes, it must be undiscovered sometimes, presumably much more often). $\endgroup$ – Glen_b Apr 2 '15 at 2:25
  • 18
    $\begingroup$ A model without assumptions is just a replication of your datamatrix. It is true, but also a useless duplication of effort. One could also argue that such a model is not a model: One definition of a model is that it is a simplification of reality. So a model that does not simplify is by definition not a model. That simplification has a purpose: it helps us substantively understand what is going on; it gives us a main story. If you don't care about understanding why things happen and only want to predict, that main story is irrelavant. However, some want to do more than create a cristal ball... $\endgroup$ – Maarten Buis Apr 2 '15 at 7:22
  • 7
    $\begingroup$ A model without assumptions is not a model. The model IS a collection of assumptions. Also, as @Glen_b says, iid assumption is rather strong, and I can't think of an example where it is "true". Even when repeatedly flipping a coin, there are correlations (see work of Diaconis). I have only skimmed the article but it seems short sighted. $\endgroup$ – P.Windridge Apr 2 '15 at 9:49
  • 11
    $\begingroup$ The author's first example about flying to Mars is bogus. Newtonian mechanics (which is relied on for many aspects of space missions) is a wrong model, but we rely on it and it is useful. $\endgroup$ – P.Windridge Apr 2 '15 at 10:20
  • 6
    $\begingroup$ Yes, Newtonian mechanics is a great example of George Box's point. $\endgroup$ – Glen_b Apr 2 '15 at 10:49
14
+150
$\begingroup$

The cited article seems to be based on fears that statisticians "will not be an intrinsic part of the scientific team, and the scientists will naturally have their doubts about the methods used" and that "collaborators will view us as technicians they can steer to get their scientific results published." My comments on the questions posed by @rvl come from the perspective of a non-statistician biological scientist who has been forced to grapple with increasingly complicated statistical issues as I moved from bench research to translational/clinical research over the past few years. Question 5 is clearly answered by the multiple answers now on this page; I'll go in reverse order from there.

4) It doesn't really matter whether an "exact model" exists, because even if it does I probably won't be able to afford to do the study. Consider this issue in the context of the discussion: Do we really need to include “all relevant predictors?” Even if we can identify "all relevant predictors" there will still be the problem of collecting enough data to provide the degrees of freedom to incorporate them all reliably into the model. That's hard enough in controlled experimental studies, let alone retrospective or population studies. Maybe in some types of "Big Data" that's less of a problem, but it is for me and my colleagues. There will always be the need to "be smart about it," as @Aksakal put it an an answer on that page.

In fairness to Prof. van der Laan, he doesn't use the word "exact" in the cited article, at least in the version presently available on line from the link. He talks about "realistic" models. That's an important distinction.

Then again, Prof. van der Laan complains that "Statistics is now an art, not a science," which is more than a bit unfair on his part. Consider the way he proposes to work with collaborators:

... we need to take the data, our identity as a statistician, and our scientific collaborators seriously. We need to learn as much as possible about how the data were generated. Once we have posed a realistic statistical model, we need to extract from our collaborators what estimand best represents the answer to their scientific question of interest. This is a lot of work. It is difficult. It requires a reasonable understanding of statistical theory. It is a worthy academic enterprise!

The application of these scientific principles to real-world problems would seem to require a good deal of "art," as with work in any scientific enterprise. I've known some very successful scientists, many more who did OK, and some failures. In my experience the difference seems to be in the "art" of pursing scientific goals. The result might be science, but the process is something more.

3) Again, part of the issue is terminological; there's a big difference between an "exact" model and the "realistic" models that Prof. van der Laan seeks. His claim is that many standard statistical models are sufficiently unrealistic to produce "unreliable" results. In particular: "Estimators of an estimand defined in an honest statistical model cannot be sensibly estimated based on parametric models." Those are matters for testing, not opinion.

His own work clearly recognizes that exact models aren't always possible. Consider this manuscript on targeted maximum likelihood estimators (TMLE) in the context of missing outcome variables. It's based on an assumption of outcomes missing at random, which may never be testable in practice: "...we assume there are no unobserved confounders of the relationship between missingness ... and the outcome." This is another example of the difficulty in including "all relevant predictors." A strength of TMLE, however, is that it does seem to help evaluate the "positivity assumption" of adequate support in the data for estimating the target parameter in this context. The goal is to come as close as possible to a realistic model of the data.

2) TMLE has been discussed on Cross Validated previously. I'm not aware of widespread use on real data. Google Scholar showed today 258 citations of what seems to be the initial report, but at first glance none seemed to be on large real-world data sets. The Journal of Statistical Software article on the associated R package only shows 27 Google Scholar citations today. That should not, however, be taken as evidence about the value of TMLE. Its focus on obtaining reliable unbiased estimates of the actual "estimand" of interest, often a problem with plug-in estimates derived from standard statistical models, does seem potentially valuable.

1) The statement: "a statistical model that makes no assumptions is always true" seems to be intended as a straw man, a tautology. The data are the data. I assume that there are laws of the universe that remain consistent from day to day. The TMLE method presumably contains assumptions about convexity in the search space, and as noted above its application in a particular context might require additional assumptions.

Even Prof. van der Laan would agree that some assumptions are necessary. My sense is that he would like to minimize the number of assumptions and to avoid those that are unrealistic. Whether that truly requires giving up on parametric models, as he seems to claim, is the crucial question.

$\endgroup$
  • $\begingroup$ Very nice answer. The book Targeted Learning is a good place to learn more. In addition to theory it contains some case studies. $\endgroup$ – NRH May 5 '16 at 22:35
12
$\begingroup$

Maybe I missed the point, but I think you have to step back a little bit.

I think his point is the abuse of easy-accessible tools with no further knowledge. This is also true for a simple t-test: just feed the algorithm with your data, getting a p<0.05 and thinking, that your thesis is true. Completely wrong. You, of course, have to know more about your data.

Stepping even further back: There is nothing like an exact model (physicist here). But some agree very well with our measurements. The only exact thing is math. Which has nothing to do with reality or models of it. Everything else (and every model of the reality) is "wrong" (as quoted so often).

But what does mean "wrong" and useful? Judge by yourself:

ALL of our current high-tech (computers, rockets, radioactivity etc) is based on these wrong models. Maybe even computed by "wrong" simulations with "wrong" models.
-> Focus more on the "useful" instead of "wrong";)

More explicitly to your questions:

  1. Don't know, sorry!
  2. Yes. One example: in particle-physics, you want to detect certain particles (say electrons, protons etc.). Every particle leaves a characteristic trace in the detector (and therefore the data), but varies even for the same particle (by its nature). Today, most of the people use machine-learning to achieve this goal (this was a huge simplification, but it is pretty much like this) and there is an increase in efficiency of 20%-50% compared to doing it by hand statistics.
  3. Nobody really claimed this! Don't make wrong conclusion! (a: all models are inexact and b: some are useful. Don't confuse things)
  4. There is no thing as an exact model (except in math, but not really in statistics as having points exactly on a straight line and "fitting" a line through it may be exact... but that's an uninteresting special case which never happens).
  5. Don't know :) But IMHO I see this more as a "just because every child can use it, not everyone should" and don't overuse it blindly.
$\endgroup$
  • $\begingroup$ I'll consider your points, but on (3) and (4), I suggest you look at van der Laan's article, as well as my question, because the article does say pretty clearly that he has no use for inexact models, and repeatedly refers to "the exact model". So yes, I'd say that somebody indeed said that. I pretty much agree with what you say about models, as well as mechanical application of methods; and so I don't think it's me you want to be asking to step back. $\endgroup$ – rvl Apr 27 '16 at 21:40
  • $\begingroup$ Yes, to all. And mostly, I meant him to step back, for sure ;) I think, what I did not mention, to build theoretical (mathematical) models, you do of course need "exact" models. So to "invent" better models and statistical tools, you need deterministic (or exact) models as a base. Even dough targeted learning does not have to be "inexact" in this sense, I think. $\endgroup$ – Mayou36 Apr 28 '16 at 12:08
6
$\begingroup$

In econ, much is said of understanding the 'data generating process.' I'm not sure what exactly is meant by an 'exact' model, but in econ it might be the same as a 'correctly specified' model.

Certainly, you want to know as much about the process that generated the data as you can before attempting a model, right? I think the difficulty comes from a) we may not have a clue about the real DGP and b) even if we knew the real DGP it might be intractable to model and estimate (for many reasons.)

So you make assumptions to simplify matters and reduce estimation requirements. Can you ever know if your assumptions are exactly right? You can gain evidence in favor of them, but IMO it's tough to be really sure in some cases.

I have to filter all of this in terms of both established theory as well as practicality. If you make an assumption consistent with a theory and that assumption buys you better estimation performance (efficiency, accuracy, consistency, whatever) then I see no reason to avoid it, even if it makes the model 'inexact'.

Frankly, I think the article is meant to stimulate those who work with data to think harder about the entire modeling process. It's clear that van der Laan makes assumptions in his work. In this example, in fact, van der Laan seems to throw away any concern for an exact model, and instead uses a mish-mash of procedures to maximize performance. This makes me more confident that he raised Box's quote with the intent of preventing people from using it as an escape from the difficult work of understanding the problem.

Let's face it, the world is rife with misuse and abuse of statistical models. People blindly apply whatever they know how to do, and worse, others often interpret the results in the most desirable way. This article is a good reminder to be careful, but I don't think we should take it to the extreme.

The implications of the above for your questions:

  1. I agree with others on this post that have defined a model as a set of assumptions. With that definition, a model with no assumptions isn't really a model. Even exploratory data analysis (i.e. model free) requires assumptions. For example, most people assume the data are measured correctly.
  2. I don't know about TMLE, per se, but in economics there are many articles that use the same underlying philosophy of inferring about a causal effect on an unobserved counterfactual sample. In those cases, however, receiving a treatment is not independent of the other variables in the model (unlike TMLE), and so economists make extensive use of modeling. There are a few case studies for structural models, such as this one where the authors convinced a company to implement their model and found good results.
  3. I think all models are inexact, but again, this term is a bit fuzzy. IMO, this is at the core of Box's quote. I'll restate my understanding of Box this way: 'no model can capture the exact essence of reality, but some models do capture a variable of interest, so in that sense you might have a use for them.'
  4. I addressed this above. In short, I don't think so.
  5. I'm not sure. I like it right here.
$\endgroup$
5
$\begingroup$

To address point 3, the answer, obviously, is no. Just about every human enterprise is based on a simplified model at some point: cooking, building, interpersonal relationships all involve humans acting on some kind of data + assumptions. No one has ever constructed a model that they did not intend to make use of. To assert otherwise is idle pedantry.

It is much more interesting and enlightening, and useful to ask when inexact models are not useful, why they fail in their usefulness, and what happens when we rely on models that turn out not to be useful. Any researcher, whether in academia or industry, has to ask that question shrewdly and often.

I don't think the question can be answered in general, but the principles of error propagation will inform the answer. Inexact models break down when the behavior they predict fails to reflect behavior in the real world. Understanding how errors propagate through a system can help one understand how much precision is necessary in modeling the system.

For example, a rigid sphere is not usually a bad model for a baseball. But when you are designing catcher's mitt, this model will fail you and lead you to design the wrong thing. Your simplifying assumptions about baseball physics propagate through your baseball-mitt system, and lead you to draw the wrong conclusions.

$\endgroup$
5
$\begingroup$

1) What useful statistical inferences can be made using a model that makes no assumptions at all?

A model is by definition a generalization of what you are observing that can be captured by certain causal factors that in turn can explain and estimate the event you are observing. Given that all those generalization algorithms have some sort of underlying assumptions. I am not sure what is left of a model if you have no assumptions whatsoever. I think you are left with the original data and no model.

2) Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?

I don't know. Maximum likelihood is used all the time. Logit models are based on those as well as many other models. They don't differ a whole lot to standard OLS where you focus on the reductions of the sum of the square of the residuals. I am not sure what targeted maximum likelihood is. And, how it differs from traditional maximum likelihood.

3) Are all inexact models indeed useless?

Absolutely not. Inexact models can be very useful. First, they contribute to better understanding or explaining a phenomenon. That should count for something. Second, they may provide descent estimation and forecasting with relevant Confidence Interval to capture the uncertainty surrounding an estimate. That can provide a lot of info on what you are studying.

The issue of "inexact" also raises the issue of the tension between parsimony and overfit. You can have a simple model with 5 variables that is "inexact" but does a pretty good job of capturing and explaining the overall trend of the dependent variable. You can have a more complex model with 10 variables that is "more exact" than the first one (higher Adjusted R Square, lower Standard Error, etc.). Yet, this second more complex model may really crash when you test it using a Hold Out sample. And, in such case maybe the "inexact" model actually performs a lot better in the Hold Out sample. This happens literally all the time in econometrics and I suspect in many other social sciences. Beware of "exact" models. They can often be synonimous with overfit models and mis-specified models (models with non stationary variables that have underlying trends (unit root) with no economic meaning imparted to the model).

4) Is it possible to know that you have the exact model other than in trivial cases?

It is not possible to know that you have the exact model. But, it is possible to know you have a pretty good model. The information criteria measures (AIC, BIC, SIC) can give you much information allowing to compare and benchmark the relative performance of various models. Also, the LINK test can also help in that regard.

5) If this is too opinion-based and hence off-topic, where can it be discussed? Because Dr van der Laan's article definitely does need some discussion.

I would think this is as appropriate a forum to discuss this issue as anywhere else. This is a pretty interesting issue for most of us.

$\endgroup$
5
$\begingroup$

(I don't see the phrase "exact model" in the article (though quoted above))

1) What useful statistical inferences can be made using a model that makes no assumptions at all?

You have to start somewhere. If that's all you have (nothing), it can be a starting point.

2) Does there exist a case study, with important, real data in the use of targeted maximum likelihood? Are these methods widely used and accepted?

To answer the second question, Targeted Maximum Likelihood turns up in 93/1143281 (~.008% ) of papers in arxiv.org. So, no is probably a good estimate (without assumptions) to that one.

3) Are all inexact models indeed useless?

No. Sometimes you only care about one aspect of a model. That aspect can be very good and the rest very inexact.

4) Is it possible to know that you have the exact model other than in trivial cases?

The best model is the model that best answers your question. That may mean leaving something out. What you want to avoid, as best you can, is assumption violation.

5) Happy hour. And drinks are cheaper to boot!

I find use of the word "exact" a bit unsettling. It's not very statistician-like talk. Inexactitude? Variation? Thank G-d! That's why we are all here. I think the phrase "All models are wrong..." is okay, but only in the right company. Statisticians understand what it means, but few others do.

$\endgroup$
  • $\begingroup$ Good point about the phrase "exact model." He says things about "true" and "actual" models that are equivalent, especially given the context and tone of the article, but you're right, the quote is not "exact" (pun intended). $\endgroup$ – SQLServerSteve May 5 '16 at 1:03
  • $\begingroup$ My bad. I should have quoted him correctly. $\endgroup$ – rvl May 5 '16 at 1:19
  • $\begingroup$ I feel that trying to achieve the true or actual model misses the point of the exercise. I think what he's really is trying to discuss is bad models. $\endgroup$ – mandata May 5 '16 at 1:27
  • $\begingroup$ Yeah the article makes many good points, but he intersperses a lot of extreme comments like "It is complete nonsense to state that all models are wrong," when they are in fact all wrong to some degree. It is nonsense to say otherwise. He's definitely staking an extreme position. Rvl was entirely right to bring this up and call him out (don't sweat the quote Ryl, the important thing is that you captured the meaning correctly). $\endgroup$ – SQLServerSteve May 5 '16 at 2:13
4
+150
$\begingroup$

Said article appears to me to be a honest but political article, a sincere polemic. As such, it contains a lot of passionate passages that are scientific non-sense, but that nevertheless may be effective in stirring up useful conversations and deliberations on important matters.

There are many good answers here so let me just quote a few lines from the article to just show that Prof. Laan is certainly not using any kind of "exact model" in his work (and by the way, who says that the "exact model" is a concept equivalent to the actual data generating mechanism?)

Quotes (bold my emphasis)

"Once we have posed a realistic statistical model, we need to extract from our collaborators what estimand best represents the answer to their scientific question of interest."

Comment: "realistic" is as removed from "exact" as the Mars is from the Earth. They both orbit the Sun though, so for some purposes it doesn't matter which planet one chooses. For other purposes, it does matter. Also "best" is a relative concept. "Exact" is not.

"Estimators of an estimand defined in an honest statistical model cannot be sensibly estimated based on parametric models...

Comment: Honesty is the best policy indeed, but it is certainly not guaranteed to be "exact". Also, "sensible estimation" appears to be a very diluted outcome if one uses the "exact model".

"In response to having to solve these hard estimation problems the best we can, we developed a general statistical approach..."

Comment: OK. We are "doing the best we can". As almost everybody is thinking about oneself. But "best we can" is not "exact".

$\endgroup$
2
$\begingroup$

I'm going to approach this from the alternate direction of philosophy, in light of the really useful principles of Uncertainty Management discussed in George F. Klir's books on fuzzy sets. I can't give van der Laan exactness, but I can provide a somewhat exhaustive case for why his goal is logically impossible; that is going to call for a lengthy discussion that references other fields, so bear with me.

Klir and his co-authors divide uncertainty into several subtypes, such as nonspecificity (i.e. when you have an unknown set of alternatives, dealt with through means like the Hartley Function); imprecision in definitions (i.e. the "fuzziness" modeled and quantified in fuzzy sets); strife or discord in evidence (addressed in Dempster-Shafer Evidence Theory); plus probability theory, possibility theory and measurement uncertainty, where the goal is to have an adequate scope to capture the relevant evidence, while minimizing errors. I look at the whole toolbox of statistical techniques as alternate means of partitioning uncertainty in different ways, much like a cookie cutter; confidence intervals and p-values quarantine uncertainty in one way, while measures like Shannon's Entropy whittle it down from another angle. What they can't do, however, is eliminate it entirely. To achieve an "exact model" of the kind van der Laan seems to describe, we'd need to reduce all of these types of uncertainty down to zero, so that there's no more left to partition. A truly "exact" model would always have probability and possibility values of 1, nonspecificity scores of 0 and no uncertainty whatsoever in the definitions of terms, ranges of values or measurement scales. There would be no discord in alternate sources of evidence. The predictions made by such a model would always be 100 percent accurate; predictive models essentially partition their uncertainty off into the future, but there would be none left to put off. The uncertainty perspective has some important implications:

• This tall order is not only physically implausible, but actually logically impossible. Obviously, we cannot achieve perfectly continuous measurement scales with infinitesimal degrees, by gathering finite observations using fallible, physical scientific equipment; there will always be some uncertainty in terms of measurement scale. Likewise, there will always be some fuzziness surrounding the very definitions we employ in our experiments. The future is also inherently uncertain, so the supposedly perfect predictions of our "exact" models will have to be treated as imperfect until proven otherwise - which would take an eternity.

• To make matters worse, no measurement technique is 100 percent free of error at some point in the process, nor can it be made comprehensive enough to embrace all of the possibly conflicting information in the universe. Furthermore, the elimination of possible confounding variables and complete conditional independence cannot be proven thoroughly without examining all other physical processes that affect the one we're examining, as well as those that affect these secondary processes and so on.

• Exactness is possible only in pure logic and its subset, mathematics, precisely because abstractions are divorced from real-world concerns like these sources of uncertainty. For example, by pure deductive logic, we can prove that 2 + 2 = 4 and any other answer is 100 percent incorrect. We can also make perfectly accurate predictions that it will always equal 4. This kind of precision is only possible in statistics when we're dealing with abstractions. Statistics is incredibly useful when applied to the real world, but the very thing that makes it useful injects at least some degree of inescapable uncertainty, thereby rendering it inexact. It is an unavoidable dilemma.

• Furthermore, Peter Chu raises additional limitations in the comments section of the article rvl linked to. He puts it better than I can:

"This solution surface of NP-hard problems is typically rife with many local optima and in most case it is computationally unfeasible to solve the problem i.e. finding the global optimal solution in general. Hence, each modeler is using some (heuristic) modeling techniques, at best, to find adequate local optimal solutions in the vast solution space of this complex objective function."

• All of this means that science itself cannot be perfectly accurate, although van der Laan seems to speak of it in this way in his article; the scientific method as an abstract process is precisely definable, but the impossibility of universal and perfect exact measurement means it cannot produce exact models devoid of uncertainty. Science is a great tool, but it has limits.

• It gets worse from there: Even if were possible to exactly measure all of the forces acting on every constituent quark and gluon in the universe, some uncertainties would still remain. First, any predictions made by such a complete model would still be uncertain due to the existence of multiple solutions for quintic equations and higher polynomials. Secondly, we cannot be completely certain that the extreme skepticism in embodied in the classic question "maybe this is all a dream or a hallucination" is not a reflection of reality - in which case all of our models are indeed wrong in the worst possible way. This is basically equivalent to a more extreme ontological interpretation of the original epistemological formulations of philosophies like phenomenalism, idealism and solipsism.

• In his 1909 classic Orthodoxy G.K. Chesterton noted that the extreme versions of these philosophies can indeed be judged, but by whether or not they drive their believers into mental institutions; ontological solipsism, for example, is actually a marker of schizophrenia, as are some of its cousins. The best that we can achieve in this world is to eliminate reasonable doubt; unreasonable doubt of this unsettling kind cannot be rigorously done away with, even in a hypothetical world of exact models, exhaustive and error-free measurements. If van der Laan aims at ridding us of unreasonable doubt then he is playing with fire. By grasping at perfection, the finite good we can do will slip through our fingers; we are finite creatures existing in an infinite world, which means the kind of complete and utterly certain knowledge van der Laan argues for is permanently beyond our grasp. The only way we can reach that kind of certainty is by retreating from that world into the narrower confines of the perfectly abstract one we call "pure mathematics." This does not mean, however, that a retreat into pure mathematics is the solution to eliminating uncertainty. This was essentially the approach taken by the successors of Ludwig Wittgenstein (1889-1951), who drained his philosophy of logical positivism of whatever common sense it had by rejecting metaphysics altogether and retreating entirely into pure math and scientism, as well as extreme skepticism, overspecialization and overemphasis on exactness over usefulness. In the process, they destroyed the discipline of philosophy by dissolving it into a morass of nitpicking over definitions and navel-gazing, thereby making it irrelevant to the rest of academia. This essentially killed the whole discipline, which had still been at the forefront of academic debate until the early 20th Century, to the point where it still garnered media attention and some of its leaders were household names. They grasped at a perfect, polished explanation of the world and it slipped through their fingers - just as it did through the mental patients GKC spoke of. It will also slip out of the grasp of van der Laan, who has already disproved his own point, as discussed below. The pursuit of models that are too exact is not just impossible; it can be dangerous, if taken to the point of self-defeating obsession. The pursuit of that kind of purity rarely ends well; it's often as self-defeating as those germophobes who scrub their hands so furiously that they end up with wounds that get infected. It's reminiscent of Icarus trying to steal fire from the Sun: as finite beings, we can have only a finite understanding of things. As Chesterton also says in Orthodoxy, "It is the logician who seeks to get the heavens into his head. And it is his head that splits."

In the light of the above, let me tackle some of the specific questions listed by rvl:

1) A model with no assumptions whatsoever is either a) not aware of its own assumptions or b) must be cleanly divorced from considerations that introduce uncertainty, such as measurement errors, accounting for every single possible confounding variable, perfectly continuous measurement scales and the like.

2) I'm still a newbie when it comes to maximum likelihood estimation (MLE), so I can't comment on the mechanics of target likelihood, except to point out the obvious: likelihood is just that, a likelihood, not a certainty. To derive an exact model requires complete elimination of uncertainty, which probabilistic logic can rarely do, if ever.

3) Of course not. Since all models retain some uncertainty and are thus inexact (except in cases of pure mathematics, divorced from real-world physical measurements), the human race would not have been able to make any technological progress to date - or indeed, any other progress at all. If inexact models were always useless, we'd be having this conversation in a cave, instead of on this incredible feat of technology called the Internet, all of which was made possible through inexact modeling.

Ironically, van der Laan's own model is a primary example of inexactness. His own article sketches out a model of sorts of how the field of statistics ought to be managed, with an aim towards exact models; there are no numbers attached to this "model" yet, no measurement of just how inexact or useless most models are now in his view, no quantification of how far we are away from his vision, but I suppose one could devise tests for those things. As it stands, however, his model is inexact. If it is not useful, it means his point is wrong; if it is useful, it defeats his main point that inexact models aren't useful. Either way, he disproves his own argument.

4) Probably not, because we cannot have complete information to test our model with, for the same reasons that we can't derive an exact model in the first place. An exact model would by definition require perfect predictability, but even if the first 100 tests turn out 100 percent accurate, the 101st might not. Then there's the whole issue of infinitesimal measurement scales. After that, we get into all of the other sources of uncertainty, which will contaminate any Ivory Tower evaluation of our Ivory Tower model.

5) To address the issue, I had to put it in the wider context of much larger philosophical issues that are often controversial, so I don't think it's possible discuss this without getting into opinions (note how that in and of itself is another source of uncertainty) but you're right, this article deserves a reply. A lot of what he says on other topics is on the right track, such as the need to make statistics relevant to Big Data, but there is some impractical extremism mixed in there that should be corrected.

$\endgroup$
  • 1
    $\begingroup$ Who are these "successors of Wittgenstein" who "destroyed the discipline of philosophy"!? The trend of post-war analytic philosophy - think of the later Wittgenstein, Austin, Quine, Kenny, Goodman, Lewis, Davidson, Rorty - seems to be rejection of the tenets of logical positivism, rehabilitation of metaphysics, & a retreat from scientism. (Rorty's comments on Nietzsche & Loyola suggest he might have agreed with Chesterton's point.) As to why Kim Kardashian's a household name rather than Saul Kripke, I suspect other trends have been at work since the early 20th Century. $\endgroup$ – Scortchi May 6 '16 at 9:28
  • $\begingroup$ These post-war reformers are unknown outside their field precisely because they came after the discipline had receded in the '20s/'30s (after a centuries-long decline the extreme logical positivists merely accelerated) into an irrelevance it has never recovered from. The damage had been already been done. By the '50s other academic disciplines were no longer looking to philosophy for leadership and have since often treated it with outright scorn, for out-of-touch, nitpicking pedantry.. I wish that were not true. The image may not reflect the reality of philosophy, but the taint still remains. $\endgroup$ – SQLServerSteve May 6 '16 at 19:05
  • $\begingroup$ I'd be happy to discuss this in chat next week if you have time, I don't want to get off on a tangent in this thread, but I suspect our positions aren't that far apart. I think you're entirely right, the movement that led to Kardashian etc. has been under way since the early 20th Century - precisely the time that the discipline of philosophy fell into eclipse (whether it deserved to is another matter; I consider it highly practical and wish it were still highly regarded). $\endgroup$ – SQLServerSteve May 6 '16 at 19:08
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Scortchi May 8 '16 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.