3
$\begingroup$

I'm going through this paper A Multi-Horizon Quantile Recurrent Forecaster. The authors state that:

3.1. Loss Function

In Quantile Regression, models are trained to minimize the total Quantile Loss (QL):

$L_q(y,\hat{y})=q(y−\hat{y})_+ +(1−q)(\hat{y}−y)_+$

where $(·)_+ = max(0, ·)$.

When q = 0.5, the QL is simply the Mean Absolute Error, and its minimizer is the median of the predictive distribution. Let K be the number of horizons of forecast, Q be the number of quantiles of interest, then the $K × Q$ matrix $\hat{Y} = [\hat{y}_{t+k}^{(q)} ]_{k,q}$ is the output of a parametric model $\boldsymbol{g(y_{:t}, x, θ)}$, e.g. an RNN. (emphasis mine; $x$ in $g(y_{:t}, x, θ)$ are the external regressors)

The model parameters are trained to minimize the total loss, $\sum_t\sum_q\sum_kL_q(y_{t+k},\hat{y}_{t+k}^{(q)})$, where t iterates through all forecast creation times (FCTs). Depending on the problem, components of the sum can be assigned different weights, to highlight or discount different quantiles and horizons.

Why is this model considered "parametric" ? I thought neural networks where by definition non-parametric?

Is it the fact that it is outputting a set of quantiles and not just a point forecast that makes it parametric? But even then - they are using the above defined Pinball Loss to calculate the quantiles, not a predefined distribution - so it still seems non-parametric to me.

How is this approach considered parametric?

$\endgroup$
  • $\begingroup$ I wonder if "parametric" just means "having parameters which are learned by some training procedure" in this usage. $\endgroup$ – Sycorax says Reinstate Monica Nov 28 '18 at 18:47
  • $\begingroup$ @Sycorax quite possible, but then what's with the $\theta$ in $g(y_{:t}, x, θ)$ ? $\endgroup$ – Skander H. Nov 28 '18 at 18:52
  • $\begingroup$ I think $\theta$ are those learned parameters. $\endgroup$ – Sycorax says Reinstate Monica Nov 28 '18 at 18:53
  • $\begingroup$ Can you provide a reference as to where its been referred to as being non-parametric? NNs typically have 1000s+ parameters (weights). I think of something like Gaussian Processes/ Bayesian models to be non-parametric. Regardless, personally, the argument of parametric vs non-parametric is fruitless. $\endgroup$ – sachinruk Aug 12 '19 at 0:40
5
$\begingroup$

A neural network is defined by the weights on its connections, which are its parameters. It doesn't matter what data the network was trained upon, once you have a set of weights, you can throw away your training dataset without repercussion. If you want to classify a new sample, you can do it with only the parameterized network. Even if your training dataset is very large, you can still describe the network with exactly the same number of parameters as a small training set.

A k-nearest neighbor classifier, as an example on the other hand, is nonparametric because it relies on the training data to make any predictions. You need every training point to describe your classifier, there's no way to abstract it into a parameterized model. In essence, the number of "parameters" in this model grows with the number of training points you have. If you want to classify a new sample, you need the training data itself, because it cannot be summarized by a smaller set of parameters. A kNN classifier training on a large dataset will have more effective parameters than a kNN classifier trained on a small dataset.

The neural network is parametric because it uses a fixed number of parameters to build a model, independent of the size of the training data.

$\endgroup$
  • $\begingroup$ This explanation makes perfect sense - but then why are neural nets frequently referred to as being non-parametric? $\endgroup$ – Skander H. Nov 28 '18 at 19:22
  • $\begingroup$ Neural networks enforce very few assumptions about the underlying distribution of the data, which tends to be a hallmark of non-parametric models. Parametric models force the data to fit into the assumed distribution, whether it's correct or not. I think some people consider NNs to be non-parametric because they can represent such a wide range of input-output mappings that they effectively don't share the distributional assumptions that are common among other parametric methods. $\endgroup$ – Nuclear Wang Nov 28 '18 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.