Interpreting parametric and non-parametric testing

I searched through questions on parametric and non-parametric test distinctions and it seems that the questions are all focused on a very specific test, data problem, or some technical distinction. I'm not interested in the issue of testing assumptions (don't; examine instead), or the issues of power or error rates.

My question is about interpretation of the two kinds of tests. Is there a distinction between how one interprets the test result between parametric and non-parametric? If you're running a non-parametric test you're weakening (eliminating) paths to discussion of the unknown population so it seems perhaps you're more limited in how you can discuss the test result. If you run a parametric test your connections to the population are conditional on the assumptions. What are the proper interpretations of each test and do these distinctions matter?

This is a welcome opportunity to discuss and clarify what statistical models mean and how we ought to think about them. Let's begin with definitions, so that the scope of this answer is in no doubt, and move on from there. To keep this post short, I will limit the examples and forgo all illustrations, trusting the reader to be able to supply them from experience.

Definitions

It looks possible to understand "test" in a very general sense as meaning any kind of statistical procedure: not only a null hypothesis test, but also estimation, prediction, and decision making, in either a Frequentist or Bayesian framework. That is because the distinction between "parametric" and "non-parametric" is separate from distinctions between types of procedures or distinctions between these frameworks.

In any event, what makes a procedure statistical is that it models the world with probability distributions whose characteristics are not fully known. Quite abstractly, we conceive of data $X$ as arising by numerically coding the values of objects $\omega\in\Omega$; the particular data we are using correspond to a particular $\omega$; and there is a probability law $F$ that somehow determined the $\omega$ we actually have.

This probability law is assumed to belong to some set $\Theta$. In a parametric setting, the elements $F\in\Theta$ correspond to finite collections of numbers $\theta(F)$, the parameters. In a non-parametric setting, there is no such correspondence. This usually is because we are unwilling to make strong assumptions about $F$.

The Nature of Models

It seems useful to make a further distinction that is rarely discussed. In some circumstances, $F$ is sure to be a fully accurate model for the data. Rather than define what I mean by "fully accurate," let me give an example. Take a survey of a finite, well-defined population in which the observations are binary, none will be missing, and there is no possibility of measurement error. An example might be destructive testing of a random sample of objects coming off an assembly line, for instance. The control we have over this situation--knowing the population and being able to select the sample truly randomly--assures the correctness of a Binomial model for the resulting counts.

In many--perhaps most--other cases, $\Theta$ is not "fully accurate." For instance, many analyses assume (either implicitly or explicitly) that $F$ is a Normal distribution. That's always physically impossible, because any actual measurement is subject to physical constraints on its possible range, whereas there are no such constraints on Normal distributions. We know at the outset that Normal assumptions are wrong!

To what extent is a not-fully-accurate model a problem? Consider what good physicists do. When a physicist uses Newtonian mechanics to solve a problem, it is because she knows that at this particular scale--these masses, these distances, these speeds--Newtonian mechanics is more than accurate enough to work. She will elect to complicate her analysis by considering quantum or relativistic effects (or both) only when the problem requires it. She is familiar with theorems that show, quantitatively, how Newtonian mechanics is a limiting case of quantum mechanics and of special relativity. Those theorems help her understand which theory to choose. This selection is usually not documented or even defended; it may even occur unconsciously: the choice is obvious.

A good statistician always has comparable considerations in mind. When she selects a procedure whose justification relies on a Normality assumption, for instance, she is weighing the extent to which the actual $F$ might depart from Normal behavior and how that could affect the procedure. In many cases the likely effect is so small that it needn't even be quantified: she "assumes Normality." In other cases the likely effect is unknown. In such circumstances she will run diagnostic tests to evaluate the departures from Normality and their effects on the results.

Consequences

It's starting to sound like the not-fully-accurate setting is hardly distinct from the nonparametric one: is there really any difference between assuming a parametric model and evaluating how reality departs from it, one the one hand, and assuming a non-parametric model on the other hand? Deep down, both are non-parametric.

In light of this discussion, let's reconsider conventional distinctions between parametric and non-parametric procedures.

• "Non-parametric procedures are robust." So, to some extent, must all procedures be. The issue is not of robustness vs non-robustness, but how robust any procedure is. Just how much, and in what ways, does the true $F$ depart from the distributions in the assumed $\Theta$? As a function of those departures, how much are the test results affected? These are basic questions that apply in any setting, parametric or not.

• "Non-parametric procedures don't require goodness-of-fit testing or distributional testing." This isn't generally true. "Non-parametric" is often mistakenly characterized as "distribution-free," in the sense of allowing $F$ to be literally any distribution, but this is almost never the case. Almost all non-parametric procedures do make assumptions that restrict $\Theta$. For instance, $X$ might be split into two sets for comparison, with a distribution $F_0$ governing one set and another distribution $F_1$ governing the other. Perhaps no assumption is made at all about $F_0$, but $F_1$ is assumed to be translated version of $F_0$. That's what many comparisons of central tendency assume. The point is that there is a definite assumption made about $F$ in such tests and it deserves to be checked just as much as any parametric assumption might be.

• "Non-parametric procedures don't make assumptions." We have seen that they do. They only tend to make less-constraining assumptions than parametric procedures.

An undue focus on parametric vs non-parametric might be a counterproductive approach. It overlooks the main objective of statistical procedures, which is to improve understanding, make good decisions, or take appropriate action. Statistical procedures are selected based on how well they can be expected to perform in the problem context, in light of all other information and assumptions about the problem, and with regard to the consequences to all stakeholders in the outcome.

The answer to "do these distinctions matter" would therefore appear to be "not really."