5
$\begingroup$

Usually a model is considered to be high-dimensional when $n \ll p$, where $n$ is the number of the observations and $p$ the number of the variables/features (e.g. Bühlmann and van de Geer, 2011). However, you can find other definitions in the literature, as in Belloni et al. (2018) Ref. They write:

High-dimensional models are characterized by having a number of unknown parameters that is not vanishingly small relative to the sample size.

So when is a model high-dimensional? Is there a rule of thumb or some criterion, which defines that a model is high-dimensional different than $n \ll p$? I know that there is similar question on StackExchange (Ref), but this post is relatively old and the field of high-dimensional statistics/ machine learning developed in the last ten years.

$\endgroup$
2
  • 4
    $\begingroup$ "High-dimensional" is merely a qualitative characterization, whose meaning will depend on the context, the data, the contemplated models, and even the date (because it will depend on computing capabilities: in 1950, five dimensions was extremely high-dimensional). Thus, don't expect any specific or general answer. BTW, if you're looking for a "criterium," attend a bicycle race! $\endgroup$
    – whuber
    Commented Jan 11, 2022 at 20:34
  • 2
    $\begingroup$ See also the Sorites paradox. $\endgroup$ Commented Jan 12, 2022 at 2:01

2 Answers 2

1
$\begingroup$

From computational point of view, the criterion for high-dimensionality of a model probably tied to curse of dimensionality. Obviously, this tied to computational resources as well (as mentioned in the comment by @whuber). The dimension $p$ where by "conventional resources" provides slow solution or becomes infeasible would be called "high dimensional model".

The case $n<<p$ would be called overparametrized model (see) rather than high-dimensional. In linear algebra, it would be called Underdetermined system. Recall LASSO.

From specific domain point of view: In physics/statistics, anything $p>3$ is high-dimensional for classical mechanics, where by in statistical mechanics $p$ could be 1000s for simulations to be called high-dimensional. See degrees of freedom. In biology/genetics "high-dimensionality" may refer to simultaneous study of different factors, see The use of high-dimensional biology.

$\endgroup$
2
  • 1
    $\begingroup$ Re physics: In classical mechanics, the dimension is the number of conjugate coordinates. A system with "moderate" dimensions might consist of a few particles to a few hundred and each particle can have $3+3=6$ position+momentum coordinates. Re computation: although your characterization is interesting and potentially useful, it seems not to distinguish "dimension" from "model complexity" or "algorithmic complexity." A model can have very few dimensions, yet be computationally difficult to fit. $\endgroup$
    – whuber
    Commented Jan 12, 2022 at 1:28
  • $\begingroup$ @whuber Agreed. A model indeed can be low dimensional with high algorithmic complexity. $\endgroup$ Commented Jan 12, 2022 at 6:50
1
$\begingroup$

An excellent introduction to the statistical aspects of this question is Chapter 1.1 in this textbook (I can recommend this book more generally). In a nutshell, a model is called high-dimensional if p (the number of parameters in the model) is of the order of n (the number of observations from that model) or larger. This definition agrees with the definitions already given, but the book makes an important distinction between high-dimensional models and high-dimensional statistics: "high-dimensional statistics specializes in models that have many parameters" and, therefore, is not limited to high-dimensional models. For example, the sparsity-inducing property of certain high-dimensional estimators (such as the lasso) can be useful for model selection when p is large—irrespective of n.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.