When is a model high-dimensional?

Question

Usually a model is considered to be high-dimensional when $n \ll p$, where $n$ is the number of the observations and $p$ the number of the variables/features (e.g. Bühlmann and van de Geer, 2011). However, you can find other definitions in the literature, as in Belloni et al. (2018) Ref. They write:

High-dimensional models are characterized by having a number of unknown parameters that is not vanishingly small relative to the sample size.

So when is a model high-dimensional? Is there a rule of thumb or some criterion, which defines that a model is high-dimensional different than $n \ll p$? I know that there is similar question on StackExchange (Ref), but this post is relatively old and the field of high-dimensional statistics/ machine learning developed in the last ten years.

Answer 1

From computational point of view, the criterion for high-dimensionality of a model probably tied to curse of dimensionality. Obviously, this tied to computational resources as well (as mentioned in the comment by @whuber). The dimension $p$ where by "conventional resources" provides slow solution or becomes infeasible would be called "high dimensional model".

The case $n<<p$ would be called overparametrized model (see) rather than high-dimensional. In linear algebra, it would be called Underdetermined system. Recall LASSO.

From specific domain point of view: In physics/statistics, anything $p>3$ is high-dimensional for classical mechanics, where by in statistical mechanics $p$ could be 1000s for simulations to be called high-dimensional. See degrees of freedom. In biology/genetics "high-dimensionality" may refer to simultaneous study of different factors, see The use of high-dimensional biology.

Answer 2

An excellent introduction to the statistical aspects of this question is Chapter 1.1 in this textbook (I can recommend this book more generally). In a nutshell, a model is called high-dimensional if p (the number of parameters in the model) is of the order of n (the number of observations from that model) or larger. This definition agrees with the definitions already given, but the book makes an important distinction between high-dimensional models and high-dimensional statistics: "high-dimensional statistics specializes in models that have many parameters" and, therefore, is not limited to high-dimensional models. For example, the sparsity-inducing property of certain high-dimensional estimators (such as the lasso) can be useful for model selection when p is large—irrespective of n.