1
$\begingroup$

Increasing the flexibility of models makes it prone to overfitting. On the other hand, it looks to me that, if the space function classes $\mathcal{F}$ is too big, it is hard to prove bounds on empirical risks bounds and that sort of stuff. That's why I am questioning the necessity/importance/applicability of non-parametric models.

Here by nonparametrics I mostly mean, Dirichlet Processes and Beta Processes (and related family).

Any comments?

$\endgroup$
2
  • 1
    $\begingroup$ I don't know why this is tagged with Bayesian - most research in nonparametric methods is frequentist. And isn't the success of the SVM an argument that we can do nonparametric stuff with provable empirical risk bounds? Neural nets are also, in spirit, nonparametric models (universal approximation theorem) with some clever regularization schemes, and these are state-of-the-art for many problems. $\endgroup$
    – guy
    Oct 30 '13 at 20:12
  • $\begingroup$ Now that you've edited, it's apparent this is about Bayesian nonparametrics, rather than the other 90% of nonparametrics. Associated with each BNP model is usually some story about the data similar to a parametric model, and they tend to do well when the story they tell makes sense. For a success story, just look at HDP; variations on HDP are very close to state-of-the-art for topic modeling. $\endgroup$
    – guy
    Oct 30 '13 at 20:58
2
$\begingroup$

Your first sentence is not necessarily correct. First off, an increase in numbers of parameters does indeed increase the degrees of freedom and the standard errors of point estimates (hence their degree of generalization). An example of this is the nested classes of Exponential and Weibull models. It's not universally agreed upon that "model complexity" necessarily means the parameter space, though, but it is a good place to start for discussion.

Semiparametric and nonparametric inference make overfitting a nonissue by generalizing the likelihood function into a new type of function where such extraneous parameters are ancillary. The only caveat is that the statistician has to correctly identify such models. Examples of such extended likelihoods are conditional likelihood (in mixed modeling), partial likelihood (in Cox models), pseudo likelihood (forgetting some applications for that...), profile likelihood, quasilikelihood, (and the list goes on). The parameter spaces for such likelihoods are seen as projections of high (possibly infinte) dimensional (compact) parameter spaces.

It's only in fully parametric inference where every causal relationship needs to be specified, such as the correlation structure for teeth within a mouth, or the correlation between failure times in a prospective cohort among denominators of individuals counted more than once. Many of these likelihoods are overly complex or intractable hence inference about them is non-existent or otherwise not popular.

Modeling processes is a fully parametric endeavor. You must be able to simulate data from an estimated data generating mechanism. SP/NP often cannot achieve this. Neither can the produce fitted effects nor can they claim to simulate realizations from any data generating process. SP/NP focuses on the point estimation of a specific parameter and efficiently calculates estimates and standard errors for that parameter cancelling out all other parameters in the data generating process through either conditioning, estimating them as nuisance parameters, or some other process.

SP/NP inference examples are the log-rank test (NP), the plain vanilla asymptotic t-test without normality assumptions (NP), conditional logistic regression (SP), generalized estimating equations (GEE), and Cox proportional hazards models (SP).

Examples where semi-parametric inference breaks down is in the case of missing at random data (as opposed to missing completely at random data), where the value of some observed outcome or covariate depends on the things which we deemed to be ancillary (such as informative censoring in Cox models). A fully likelihood based survival analysis would require separate models (and their correlation) for survival and censoring outcomes.

$\endgroup$
14
  • $\begingroup$ This is a very classical perspective. Personally, I'd be fine with calling RBMs or ANNs "nonparametric" but you certainly can simulate from them. Ditto for Bayesian nonparametric methods. $\endgroup$
    – guy
    Oct 30 '13 at 20:49
  • $\begingroup$ You have a source or reasoning for why those machine learning techniques are nonparametric? They're not really used for inference, so I don't even see how that enters the equation. $\endgroup$
    – AdamO
    Oct 30 '13 at 20:56
  • $\begingroup$ Bayesian nonparametric has nonparametric right there in the name (formally, we put a prior on the space of cdfs which has large support) and is used for inference all the time. To simulate we can just draw the measure from the posterior. OP is also likely from ML given that he is using Statistical Learning Theory lingo. $\endgroup$
    – guy
    Oct 30 '13 at 21:02
  • $\begingroup$ The ML models can be formalized as nonparametric using sieves (which to an extent is similar to how they are used) I think, combined with universal approximation I'd say nonparametric is a fine label. $\endgroup$
    – guy
    Oct 30 '13 at 21:07
  • 1
    $\begingroup$ "Nonparametric" doesn't typically mean "I have no parameters." Even in the simplest case of $X_1, \ldots, X_n \sim F$ where $F$ is an unspecified cdf, $F$ is just a parameter in the (infinite-dimensional) space of cdfs. A common definition of a nonparametric problem is one with an infinite-dimensional parameter space, and a method is nonparametric if it can estimate anything in that space. Then, the class of ANNs (with network topology a free parameter) fits this definition for nonparametric function estimation. $\endgroup$
    – guy
    Oct 30 '13 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.