Skip to main content

Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Bound of operator norm for Gaussian ensemble (wainwright example 6.2)

Consider $W \in \mathbb{R}^{n \times d}$ generated with i.i.d. $N(0,1)$ entries, theorem 6.1 in the Martin Wainwright HDS implies that $$ \frac{\sigma_\max(W)}{\sqrt{n}} \leq 1 + \delta + \sqrt{\frac{...
Mondayisgood's user avatar
1 vote
0 answers
13 views

Sampling from a hypersphere subject to a linear constraint? [duplicate]

I'm running into efficiency issues when trying to sample from a "hypercone" using rejection sampling. By a hypercone, I mean the set of vectors $C_{v,\beta} = \{w \sim N(0,1)\ |\ w^T v \geq \...
billybobsteve's user avatar
4 votes
1 answer
45 views

The sum of $O_p$ --$ O_p \left(s^2\frac{\log d}{n}+s\sqrt{\frac{\log d}{n}} \right) $

I read papers in the area of inference for high-dimensional graphical models and these papers always state the convergence rate of the estimator. Using $O_p$ is a good choice. Maybe I made some ...
mathhahaha's user avatar
1 vote
0 answers
39 views

how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?

Context I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
Tabitha Peter's user avatar
0 votes
0 answers
21 views

Wide Shallow Neural Networks VS Deep NN

I have several key points of understanding, but I cannot reach a final conclusion on why shallow neural networks cannot model data as effectively as deep neural networks. I understand that we can ...
rando's user avatar
  • 308
6 votes
1 answer
111 views

Bound on Rademacher complexity using polynomial discrimination

This is lemma 4.14 in Wainwright's textbook on High-Dimensional Statistics, it states that given a class of function $\mathcal{F}$ has polynomial discrimination of order $v$, then for all integer $n$ ...
Mondayisgood's user avatar
0 votes
1 answer
34 views

High dimensional regression with millions of covariates/features

as a matter of preamble, I am a machine learning researcher. I am interested if this community can point me to research and work showing settings that have performed regression where the number of ...
adebayoj's user avatar
1 vote
1 answer
23 views

The covering number of a d-dim cube

In Martin Wainwright's textbook, equation (5.5) states that the $\delta$-covering number of the d-dimensional cube satisfies $$ \log N(\delta; [0,1]^d) \asymp d \log(\frac{1}{\delta}), $$ for small ...
Mondayisgood's user avatar
2 votes
1 answer
54 views

Should we routinely conduct unsupervised learning when reporting descriptive statistics on data?

A standard approach prior to conducting a predictive or inferential analysis is to report some basic univariate descriptive statistics on the study variables: mean, median, minimum, maximum, variance, ...
RobertF's user avatar
  • 6,194
0 votes
1 answer
44 views

What is the meaning of $\asymp$ and $\lesssim$ in Martin wainwright's high dim textbook? [closed]

Unfortunately, this text book did not provide a table of notations he used. Can anyone provide me with a definition of $\asymp$ and $\lesssim$ and few examples? For an example in the book, in display (...
Mondayisgood's user avatar
1 vote
1 answer
47 views

Modeling a high dimensional multicollinear data

I am trying to predict a Plant physiology trait (y) from hyper spectral reflectance data from 400 to 2400nm (X). So far i have done the following Skew correction with Square root (sqrt) on y Scaling ...
Aaron Poruthoor's user avatar
1 vote
0 answers
61 views

Maximum Likelihood in High Dimensions [closed]

What are some examples of high-dimensional random variables for which MLE are solved using numerical methods because we are unable to explicitly solve the equations nicely? The only example to comes ...
Nicolas Bourbaki's user avatar
1 vote
0 answers
20 views

Large $N$, small $T$ in SUR: workaround using system GMM

Consider a system of linear equations as in seemingly unrelated regression (SUR). If the number of equations $N$ is large relative to the sample size $T$, the weighting matrix in SUR (i.e. the error ...
Richard Hardy's user avatar
1 vote
0 answers
16 views

Expected value of Cosinus in High dimension

I would like to prove that the cosinus of the angle formed by 3 randomly points tends to $\frac{1}{2}$ as the dimensionality tends to $\infty$. Could it be solved with the expected value formula ? It ...
Jérémy's user avatar
1 vote
0 answers
21 views

Benjamini Hochberg Procedure [closed]

I am working on a problem for class related to multiple testing where I would like to run the BH procedure with a known $\pi_{0}$, denoting the proportion of hypothesis that are truly null, given ...
Harry Lofi's user avatar
3 votes
1 answer
176 views

Explanation of the proof that SCAD penalty has the oracle property

I am trying to understand the proof that the SCAD has the oracle property. Could you help me with an explanation and a full break down of the steps, so that I can understand it? I'm unclear on how ...
mike's user avatar
  • 31
3 votes
1 answer
37 views

How do I interpret a second-order multi-variate growth model?

I am running a multi-variate second order growth model. I have two factors, which are conceptually related to each other measured on 7 different occasions. Wanting to know how the two factors ...
EmH's user avatar
  • 315
0 votes
0 answers
26 views

Why does FPCA not use scaling as PCA?

Functional principal component analysis (FPCA), according to the original paper, does not use scaling before FPCA, as in PCA. Instead, it uses a covariance matrix to compute the eigen-components. I ...
Palantir's user avatar
8 votes
1 answer
296 views

Regression At Scale: Best Practices Around Ensuring Quality of a Large Numbers of Forecasts

Background Often I am forecasting possibly one up to a few dozen variables in a project, but I have an upcoming project that will involve forecasting thousands of variables. I have some ideas of my ...
Galen's user avatar
  • 9,421
1 vote
0 answers
51 views

Concentration around the median implies concentration around the mean [duplicate]

Let $M$ denote the median of a function $f(X)$ that is Lipschitz continuous with $\left \| f \right \|_{Lip}=1$. I am trying to show that if $\left \| f(X)-M \right \|_{\psi_{2}}\leq C$, then $\left \|...
Shawn Kemp's user avatar
0 votes
0 answers
55 views

Independent features but PCA improves classifiers accuracy significantly. Why?

that's my first question on here :) I am working with the kNN classifier on datasets from the multivariate normal distribution. I have to groups coming from ...
Superintendant's user avatar
1 vote
0 answers
89 views

Efficient way to compute covariance matrix of Vector Autoregressive Process of order 1 (VAR)

For a VAR process $$ X_t = A_1 X_{t-1} + \epsilon_t $$ The covariance of $X_t$ can be computed in the following way: $$ \text{vec}(\Sigma) = (I -(A \otimes A))^{-1} \text{vec}(\Sigma_{\epsilon}) $$ ...
Dylan Dijk's user avatar
3 votes
1 answer
61 views

Selecting variables using lasso algorithm

I have a question concerning a large dataset with 94 observation and 15000 variables. For data mining models (boosting, trees, neural networks...) this number of variables are too much and I have to ...
ali's user avatar
  • 71
0 votes
0 answers
17 views

what does edge mean in the context of sample space?

This is a follow-up question on this post It uses a term 'edge'. For example, it says I understand that extrapolation is harder than interpolation. And I understand that if we choose a point to ...
Sherlock_Hound's user avatar
0 votes
0 answers
10 views

Lasso Screenings in biglasso package

I am currently using the biglasso package for some simulations. For the biglasso() function, there is an argument called screening. The default screening for lasso is "adaptive". Is adaptive ...
jercai's user avatar
  • 1
0 votes
1 answer
143 views

Choosing a probability distribution for 4D data: dirichlet challenges and alternatives

I'm seeking the right distribution for my 4D data, where the sum of values in each sample equals one. Currently, I've chosen to employ the Dirichlet distribution. However, upon applying this ...
roan's user avatar
  • 1
0 votes
1 answer
78 views

What exactly is the KKT check and what is the point of it?

In the paper for strong screening rules for the lasso (link), the following screening algorithm is proposed (start of chapter 7): Let $S(\lambda)$ be the strong rule set. Then the following strategy ...
Sparsity's user avatar
  • 584
0 votes
0 answers
60 views

Comparing scree plots or explained variance of two groups with different number of features after PCA

I want to define the dimensionality of a group as the number of PC features that can explain 80% of the variance in the group dataset. This intuition seems to work for a single group, however, if I ...
Jules's user avatar
  • 1
1 vote
2 answers
139 views

Tensorization of entropy: confusion regarding conditional entropy

I'm reading High-Dimensional Statistics by Wainright. In the book, entropy for random variable $Z \geq 0$ is defined as $H(Z) = E[Z \log Z]- E[Z] \log E[Z]$. My understanding is that $H(Z)$ is a ...
Phil's user avatar
  • 636
0 votes
1 answer
374 views

Seeking recommendations for feature selection methods before applying a random forest model to high-dimensional data

I'm seeking recommendations for feature selection methods before applying a random forest model to high-dimensional data, specifically with over 60,000 features and only 1,000 samples. My concern is ...
Meow Mix's user avatar
0 votes
0 answers
52 views

Do neural networks work with high-dimensional small sample data?

Are neural networks used for the classification of high-dimensional small sample data like gene microarray data, where the dimension may be on the order of 1000 features per sample with around 100 ...
Undertherainbow's user avatar
1 vote
0 answers
57 views

Clustering or factor analysis for dimensionality reduction in multivariate linear regression

I have dataset describing aggregated purchases from multiple brands. It contains variables: Brand (ordinal) Promotion (ordinal) Sales (numeric) I need to use linear regression to describe the effect ...
Lazy Artist SQuex's user avatar
3 votes
1 answer
105 views

How to show the existence of global minimizer of Lasso type of objective function?

Suppose the objective function to be minimized is $$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$ where $\theta$ is the independent variable which is feasible in $\mathbb{R}^p$...
Zifeng Zhang's user avatar
1 vote
0 answers
25 views

With mult-dimensional input vectors, what are the dimensions of the covariance matrix elements? (Gaussian Process)

I am trying to create a Bayesian Optimisation code with a Gaussian Process. My input data, $\vec{X}_i$ is 8-dimensional, where each dimension corresponds to a feature of my data, $\vec{X}_i = [\...
shadowbiscuit's user avatar
2 votes
0 answers
91 views

Why do we need $\gamma>2$ in SCAD penalty?

The SCAD penalty $p(x | \lambda, \gamma)$ from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ...
Zifeng Zhang's user avatar
1 vote
3 answers
114 views

How best to regularize high-cardinality fixed effects?

Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ...
StatStudent19's user avatar
0 votes
1 answer
25 views

When a non-subgaussian vector has subgaussian components

The following remark is from Asymptotically Normal and Efficient Estimation of Covariate-Adjusted Gaussian Graphical Model by Chen et al. 2016: Here, $X$ is a design matrix with iid rows and $X^{(1)}$...
jack of all woes's user avatar
3 votes
1 answer
115 views

Inference for high dimensional models based on running a (G)LM on the union of selected variables across best subset fits on bootstrapped datasets

I am in the process of developing an R package for best subset selection, which approximates the best subset using an iterative adaptive ridge regression procedure (with the weighted least squares ...
Tom Wenseleers's user avatar
1 vote
1 answer
133 views

Example of Failure of Hoeffding's Inequality for Empirical Risk Minimization

I am studying Introduction to Statistical Learning Theory by Bousquet, Boucheron and Lugosi. On pages 183 through 185 it considers the applicability of Hoeffding's Inequality to Empirical Risk ...
Extrava's user avatar
  • 123
0 votes
1 answer
246 views

Can I apply PCA to combine correlated variables into one variable? [closed]

I have a dataset with 11 observations and 11 features. I want to use linear regression for estimating the coefficients by using OLS method. I know it is not advisable to use linear regression with ...
Davie Blain's user avatar
9 votes
1 answer
326 views

On the high dimensional bootstrap

El Karoui and Purdom wrote a mathematically solid paper on how the bootstrap as a general resampling technique fails in high dimensions: https://arxiv.org/abs/1608.00696. I think it is a very ...
Landon Carter's user avatar
0 votes
0 answers
59 views

Class-Imbalance: How to handle different class distributions in training and held-out test data?

My dataset is high dimensional (sample size is 200 with 300 features) and imbalanced. The imbalance ratio is 80:20 in the training set and 88:12 in the held-out test set (collected at a different time ...
Dushi Fdz's user avatar
  • 145
4 votes
3 answers
3k views

A way to train a model on data with a very large number of features

I have standard data: where rows are observations, and columns are features. ...
mr.T's user avatar
  • 269
1 vote
0 answers
25 views

Simulation case for Double machine learning in Partial linear model with lasso where number of covariates is large

I want to conduct a simulation study on double machine learning in Partial linear regression setup as described in Chernozukov's paper for Double machine learning. I want to use lasso and want the ...
Dataved's user avatar
  • 11
1 vote
0 answers
840 views

How can I compute the SHAP values in a high dimensional dataset?

I have a classiification problem with a dataset where the number of variables is very large, and the number of observations is small. Approximately 200 observations and 10000 variables. I am using a ...
Alberto Perez Martinez's user avatar
0 votes
0 answers
18 views

Is there any theory pertaining to an assumption to would allow the following inequality to hold (constrained LASSO)

Suppose we observe the vector-matrix pair $(y,\mathbf{X})\in\mathbb{R}^n\times\mathbb{R}^{n\times d} $ which is linked by the observation model: \begin{equation} y=\mathbf{X}\theta^*+\epsilon \end{...
Carl's user avatar
  • 1,226
1 vote
0 answers
18 views

Does min-norm least squares solve regular least squares in some basis?

For a data matrix $X$ of dimension $n \times p$ where $p > n$ and corresponding label vector $y$ of dimension $n$, the standard least squares fit, $\hat{\beta} = (X^TX)^{-1}X^Ty$, is ...
Seraf Fej's user avatar
  • 556
2 votes
1 answer
66 views

Is kernel density estimation sub-gaussian?

Let $X_1, ..., X_n$ be i.i.d. samples drawn from a pdf $f(x)$ on the real line. The kernel density estimator is defined as follows, $$\hat{f_n}(x) = \frac{1}{nh}\sum_1^n K(\frac{x-X_k}{h})$$ where $K:\...
dc3506's user avatar
  • 65
1 vote
0 answers
36 views

Can I only use cross-validation when sample size is very small or do I still need a held-out test set? [duplicate]

I am trying to predict a binary outcome. My sample size is very small (n=160) and has a high-class imbalance (80:20). All the variables are highly correlated, and the dataset is high dimensional (the ...
Dushi Fdz's user avatar
  • 145
4 votes
0 answers
61 views

How can it be that the volume of a d dimensional ball is concentrated on two different equators?

In the answer to Explanation for this event on a high-dimensional dataset it is stated that: "almost all the surface area of a sphere in d -dimensional Euclidean space Ed is concentrated around ...
Dan Beiser's user avatar

1
2 3 4 5
9