Questions tagged [high-dimensional]
Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)
386
questions
0
votes
0
answers
20
views
Comparing scree plots or explained variance of two groups with different number of features after PCA
I want to define the dimensionality of a group as the number of PC features that can explain 80% of the variance in the group dataset. This intuition seems to work for a single group, however, if I ...
- 1
0
votes
2
answers
80
views
Tensorization of entropy: confusion regarding conditional entropy
I'm reading High-Dimensional Statistics by Wainright.
In the book, entropy for random variable $Z \geq 0$ is defined as $H(Z) = E[Z \log Z]- E[Z] \log E[Z]$. My understanding is that $H(Z)$ is a ...
- 554
0
votes
1
answer
63
views
Seeking recommendations for feature selection methods before applying a random forest model to high-dimensional data
I'm seeking recommendations for feature selection methods before applying a random forest model to high-dimensional data, specifically with over 60,000 features and only 1,000 samples. My concern is ...
- 1
0
votes
0
answers
26
views
Do neural networks work with high-dimensional small sample data?
Are neural networks used for the classification of high-dimensional small sample data like gene microarray data, where the dimension may be on the order of 1000 features per sample with around 100 ...
- 113
1
vote
0
answers
53
views
Clustering or factor analysis for dimensionality reduction in multivariate linear regression
I have dataset describing aggregated purchases from multiple brands. It contains variables:
Brand (ordinal)
Promotion (ordinal)
Sales (numeric)
I need to use linear regression to describe the effect ...
3
votes
0
answers
39
views
How to show the existence of global minimizer of Lasso type of objective function?
Suppose the objective function to be minimized is
$$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$
where $\theta$ is the independent variable which is feasible in $\mathbb{R}^p$...
- 71
1
vote
0
answers
18
views
With mult-dimensional input vectors, what are the dimensions of the covariance matrix elements? (Gaussian Process)
I am trying to create a Bayesian Optimisation code with a Gaussian Process. My input data, $\vec{X}_i$ is 8-dimensional, where each dimension corresponds to a feature of my data,
$\vec{X}_i = [\...
- 111
0
votes
0
answers
35
views
How to deal with the high dimensionality when using EM algorithm to solve Gaussian mixture models?
When I use the EM algorithm to solve a Gaussian mixture model, we may encounter the computation of Gaussian densities in the E step. Specifically, we should have the posterior probability as
$$
\pi_{...
- 1
0
votes
0
answers
27
views
Concentration inequality for subgaussian random vector
If I have a vector $Z $ in $\mathbb R_d $ space and a vector $B$ in $\mathbb R_d$ space, $Z $ having independent Subgaussian coordinates, is $Z^\top B$ a Subgaussian random variable?
I know sub-...
2
votes
0
answers
54
views
Why do we need $\gamma>2$ in SCAD penalty?
The SCAD penalty $p(x | \lambda, \gamma)$ from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ...
- 71
1
vote
3
answers
74
views
How best to regularize high-cardinality fixed effects?
Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ...
- 331
0
votes
1
answer
18
views
When a non-subgaussian vector has subgaussian components
The following remark is from Asymptotically Normal and Efficient Estimation of Covariate-Adjusted Gaussian Graphical Model by Chen et al. 2016:
Here, $X$ is a design matrix with iid rows and $X^{(1)}$...
3
votes
1
answer
94
views
Inference for high dimensional models based on running a (G)LM on the union of selected variables across best subset fits on bootstrapped datasets
I am in the process of developing an R package for best subset selection, which approximates the best subset using an iterative adaptive ridge regression procedure (with the weighted least squares ...
- 3,021
1
vote
1
answer
58
views
Example of Failure of Hoeffding's Inequality for Empirical Risk Minimization
I am studying Introduction to Statistical Learning Theory by Bousquet, Boucheron and Lugosi. On pages 183 through 185 it considers the applicability of Hoeffding's Inequality to Empirical Risk ...
- 123
0
votes
0
answers
17
views
Using Dudley Integral to estimate maximum singular value of Gaussian random matrices [duplicate]
On Exercise 5.14 of Wainwright, it provides a way to estimate maximum singular value of Gaussian random matrices using the one-step discretization bound and Gaussian comparison inequality.
Can we use ...
- 65
0
votes
1
answer
96
views
Can I apply PCA to combine correlated variables into one variable? [closed]
I have a dataset with 11 observations and 11 features. I want to use linear regression for estimating the coefficients by using OLS method.
I know it is not advisable to use linear regression with ...
- 25
8
votes
1
answer
232
views
On the high dimensional bootstrap
El Karoui and Purdom wrote a mathematically solid paper on how the bootstrap as a general resampling technique fails in high dimensions: https://arxiv.org/abs/1608.00696.
I think it is a very ...
- 1,545
0
votes
0
answers
38
views
Class-Imbalance: How to handle different class distributions in training and held-out test data?
My dataset is high dimensional (sample size is 200 with 300 features) and imbalanced. The imbalance ratio is 80:20 in the training set and 88:12 in the held-out test set (collected at a different time ...
- 145
4
votes
3
answers
1k
views
A way to train a model on data with a very large number of features
I have standard data: where rows are observations, and columns are features.
...
- 249
1
vote
0
answers
19
views
Simulation case for Double machine learning in Partial linear model with lasso where number of covariates is large
I want to conduct a simulation study on double machine learning in Partial linear regression setup as described in Chernozukov's paper for Double machine learning. I want to use lasso and want the ...
- 11
1
vote
0
answers
300
views
How can I compute the SHAP values in a high dimensional dataset?
I have a classiification problem with a dataset where the number of variables is very large, and the number of observations is small. Approximately 200 observations and 10000 variables. I am using a ...
0
votes
0
answers
18
views
Is there any theory pertaining to an assumption to would allow the following inequality to hold (constrained LASSO)
Suppose we observe the vector-matrix pair $(y,\mathbf{X})\in\mathbb{R}^n\times\mathbb{R}^{n\times d} $ which is linked by the observation model:
\begin{equation}
y=\mathbf{X}\theta^*+\epsilon
\end{...
- 1,144
0
votes
0
answers
26
views
Applying non-linear dimensionality reduction on binary data
Would be beneficial to apply nonlinear dimensionality reduction on a binary dataset (around 200 binary features) ? what is difference from applying MCA (multiple correspondence analysis).
- 11
1
vote
0
answers
17
views
Does min-norm least squares solve regular least squares in some basis?
For a data matrix $X$ of dimension $n \times p$ where $p > n$ and corresponding label vector $y$ of dimension $n$, the standard least squares fit, $\hat{\beta} = (X^TX)^{-1}X^Ty$, is ...
- 546
2
votes
1
answer
45
views
Is kernel density estimation sub-gaussian?
Let $X_1, ..., X_n$ be i.i.d. samples drawn from a pdf $f(x)$ on the real line. The kernel density estimator is defined as follows,
$$\hat{f_n}(x) = \frac{1}{nh}\sum_1^n K(\frac{x-X_k}{h})$$
where $K:\...
- 65
1
vote
0
answers
34
views
Can I only use cross-validation when sample size is very small or do I still need a held-out test set? [duplicate]
I am trying to predict a binary outcome. My sample size is very small (n=160) and has a high-class imbalance (80:20). All the variables are highly correlated, and the dataset is high dimensional (the ...
- 145
4
votes
0
answers
52
views
How can it be that the volume of a d dimensional ball is concentrated on two different equators?
In the answer to Explanation for this event on a high-dimensional dataset it is stated that: "almost all the surface area of a sphere in d
-dimensional Euclidean space Ed is concentrated around ...
- 41
0
votes
0
answers
33
views
Sampling and backwards selection
I'm working on a school project that involves performing backward stepwise regression as a form of feature selection. The dataset in question is 60k images with 700 total columns and is much too large ...
- 329
1
vote
0
answers
60
views
Lasso regularization doesn't work on high dimensions?
Please see the following in highlighted (ISLR 2nd Edition pg 265):
The last sentence (p=2000 example) is very concerning to me. I was going to use Lasso in ...
- 704
0
votes
0
answers
16
views
Example for number of predictors increasing with sample size
High-dimensional literature typically assumes that the number of predictors $p$ increases with sample size $n$, say $p = O(\exp n)$. I can understand that assuming $n\to\infty$ corresponds to ...
0
votes
0
answers
14
views
Quantifying the amount of information of a temporal signal after a kernel trick
We commonly use kernel trick, like in Neural Net, Support Vector Machine, etc, to map input signal to a high-dimensional feature spaces. Such mapping has enabled us to perform non-linear ...
- 1
0
votes
0
answers
69
views
Multiple linear regression machine learning can work well with high-dimensional data?
My data and problem:
Data is high-dimensional data with 15474 features (columns) and 375 cases (observations, rows). Data contains 34% zero values.
Output/response is multiple outputs of 2 (...
1
vote
0
answers
36
views
How can I recreate the lasso contour plots? [closed]
I'm trying to recreate a lasso contour plot, like either of these:
I've written this R code for it:
...
- 75
2
votes
0
answers
28
views
Proof of identification of causal effect in high dimensional mediation analysis
I'm studying this paper for high-dimensional mediation analysis for a class project. Section 2 of this paper talks about different assumptions being made for identifying causal effects. But my ...
- 3,525
1
vote
0
answers
34
views
Fitting an intercept with glmnet
I've been looking through this answer as to the behaviour of glmnet with an intercept. I've found other penalised packages do something similar, where the intercept is calculated at the end (after ...
- 75
4
votes
1
answer
414
views
Why do I get a non-zero intercept using the lasso even though I centered the response?
It's my understanding that if I center $y$, the intercept should be 0. However, when using glmnet, I get a non-zero intercept doing this:
...
- 75
0
votes
0
answers
160
views
What is better OMP or LASSO?
Variable selection in linear regression models is quite important. In this regard, the orthogonal matching pursuit (OMP) is a classical greedy approach to variable selection. On the other hand, LASSO ...
- 31
1
vote
0
answers
35
views
Random forests in low-dimensional vs high-dimensional datasets
Summary: Is random forest naturally more robust to bad features in high vs low dimensional datasets?
I've made a number of text classifiers with Scikit's Random Forest, with thousands of features from ...
- 11
2
votes
1
answer
241
views
Curse of dimensionality using trees
The curse of dimensionality refers to the fact when a model tries to fit the data in a very high dimensional space (and there is not enough training data). In my mind, I believe that this curse ...
- 227
5
votes
1
answer
88
views
Additivity of angles in higher dimensions
Suppose $x_i$'s are IID random samples from $d$-dimensional isotropic Gaussian centered at zero and normalized to have $E[\|x\|^2]=1$
Suppose we have
$$a=x_1 \\ b=a+x_2 \\ c=b+x_3$$
As $d$ increases, ...
- 6,107
4
votes
0
answers
61
views
The probability limit of the inverse of an infinite-dimensional matrix
I am considering a question regarding the calculation of the probability limit for a high-dimensional inverse matrix.
Specifically, suppose that $A_n, B_n \in \mathbb{R}^{N_n \times N_n}$ where $N_n \...
- 105
8
votes
1
answer
266
views
Does $E\frac{1}{\|x\|^{4}} \rightarrow \frac{1}{E\|x\|^4}$ in high dimensions?
Suppose we have a Gaussian distribution centered at zero, covariance matrix $\Sigma$ with $\operatorname{Tr}\Sigma=1$ and $\operatorname{Tr}\Sigma^2=\frac{1}{2}$
When I try various sequences of such ...
- 6,107
1
vote
0
answers
27
views
Why do we need exponentially more data for an accurate prediction when working in a higher dimensional space? [duplicate]
I am a bit confused by the phenomenon of the curse of dimensionality. Most lecturers motivate this with the KNN classifier and I understand why higher dimensions should be avoided with this classifier ...
- 61
1
vote
0
answers
38
views
How can I check if the curse of dimensionality negatively affects my clustering results
I would like to cluster 80 days using k-means. Each of the 80 days contains 4 time series (temperature, solar radiation, electricity demand, electricity price) with 288 values each. So all in all I ...
- 312
1
vote
1
answer
51
views
high-dimensional GLS
I'm looking for a fast and stable method to compute high-dimensional GLS estimator.
Specifically, let $\mathbf{A}$ be a $p^2 \times m$ matrix with full column rank ($rank(\mathbf{A})=m$), $\mathbf{H}$ ...
- 345
2
votes
1
answer
68
views
Coordinates of the lasso estimator
The lasso estimator is
$$
\hat\beta = \underset{\beta}{\text{argmin}}||Y-X\beta||_2^2+r||\beta||_1
$$
I always read that the coordinates $\hat\beta_j$ of the lasso estimator tend to be either clearly ...
- 23
2
votes
2
answers
278
views
Textbook on high-dimensional statistics
I am a beginning PhD student in biostatistics and want to learn about high-dimensional statistics. I have looked into the books by Buehlmann/Geer, Wainwright, and Giraud, but they seem to be targeted ...
1
vote
1
answer
140
views
Mass around mode in high dimensional Gaussian
I came across this quote in this paper. The paper is about training a classfier on images with added Gaussian noise $\delta \sim N(x, \sigma^2I)$. The paper states:
In high dimension, the Gaussian ...
- 833
5
votes
1
answer
82
views
Expected length of vectors after orthogonalization
Suppose I take $k$ vectors randomly sampled from surface of unit sphere in $d$ dimensions.
$$v_1, v_2, v_3,\ldots ,v_k$$
I apply Gram-Schmidt orthogonalization (but not orthonormalization) to obtain ...
- 6,107
1
vote
0
answers
23
views
Tests for Equality of Distribution for High Dimensional Outcomes
Suppose I have two groups, $A$ and $B$, and a null hypothesis which states that the distribution of some high-dimensional random vector $\mathbf X$ is equal across the two groups, i.e. $\mathbf X | A \...
- 2,367