Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [high-dimensional]

Pertains to great number of features or many dimensions (variables) for data points. (For great number of data points, use the tag 'large-data'.)

0
votes
0answers
6 views

Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...
2
votes
1answer
24 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
8 views

is random projection a linear or non-linear feature extraction method?

The dimensionality reduction has two different types: feature selection and feature selection. As far as i know, the random projection cannot be a feature selection method. Therefore, is it a linear ...
0
votes
0answers
29 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
1
vote
1answer
23 views

State space with lasso

Is it possible to incorporate lasso variable selection in the high dimensional state space model. If yes, is there any code or package available in R
0
votes
1answer
21 views

Euclidean distances in spaces of different dimensions

A modest attempt to illustrate my question: Setup Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, ...
0
votes
1answer
24 views

Application of LASSO , Ridge, PLS in feature selection of spectral data

The meatspec data in faraway package is spectral data with 100 features .(215 *101). Use of LASSO over ridge and PLS gives better performance (RMSE based) But none of the features are removed ( no ...
0
votes
0answers
14 views

Implementation of Compressed Regression (examples?)

I found several articles discussing compressed regression, whether in the bayesian framework or ordinary LS/GLM (1, 2, 3 and others). The idea seems simple, the model becomes $$ Y = \Phi X \beta + \...
0
votes
0answers
33 views

Main differences (problems and mathematics) between traditional statistics and high dimensional statistics

High dimensional statistics seems to be hot nowadays. What are the main differences, in terms of questions and problems it tries to solve, as well as the mathematical tools used, between "traditional"...
1
vote
2answers
26 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
1
vote
0answers
27 views

t-SNE for finding nearest neighbors

I had a question about dimensionality reduction for finding nearest neighbors and was hoping someone could help me out here. Suppose I have good features for images, say penultimate layer features ...
-1
votes
0answers
20 views

Asymptotics for Lasso-Type Estimators

In the paper Asymptotics for Lasso-Type Estimators, https://projecteuclid.org/euclid.aos/1015957397 the authors study the asymptotic properties of the LASSO estimators. I am confused, how can we ...
9
votes
1answer
165 views

How do children manage to pull their parents together in a PCA projection of a GWAS data set?

Take 20 random points in a 10,000-dimensional space with each coordinate iid from $\mathcal N(0,1)$. Split them into 10 pairs ("couples") and add the average of each pair ("a child") to the dataset. ...
1
vote
0answers
11 views

margin point of view of logistic regression

I am currently studying Lasso for logistic regression and am using Buhlmann et.al book (Statistics for High Dimensional Data) to understand better. Section In this book on page 3.3.1 they define ...
3
votes
0answers
62 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
0
votes
1answer
25 views

Classification problem for continous target

I have a big dataset with around 100k samples and 2k real-value features. The target variable is in [-1,1], but its distribution is highly concentrated around zero (...
-1
votes
0answers
48 views

Multi-dimensional Gaussian process regression

Are there extensions for Gaussian process regression (GPR) from the one-dimensional case to examples where GPR can handle multi-dimensional inputs and/or outputs? If so, could you refer me to basic ...
1
vote
1answer
106 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
0
votes
0answers
11 views

non-normal censored outcome, high dimensional predictors

I have an outcome that is non-normal (multimodal) and has an upper limit of detection giving censored values at an upper bound. I have several hundred observations and wish to predict these non-normal ...
0
votes
0answers
12 views

Are there any plots for the results of the Lasso estimator besides plotting the Lasso path?

When one reports the results of methods like Lasso, group Lasso or Stability Selection, are there any nice plots one could generate for genome-wide association studies (besides lasso paths) to make ...
2
votes
0answers
27 views

PCA: mean of marginal distribution of high-dimensional vector

Consider the following probabilistic model: $$p(x) = \mathcal{N}(0, I_d), \ x \in \mathbb{R}^d$$ $$p(y|x) = \mathcal{N}(Wx + \mu, \sigma^2I_D), \ y, \mu \in \mathbb{R}^D, W \in \mathbb{R}^{D\times d}...
1
vote
0answers
24 views

algorithms for high dimensional data in a small sample size

I am trying to analysis high-dimensional data in a small sample size and I meet a problem: First,I divided all samples into two sets: training set and test set. Second, in the training set The Y is ...
0
votes
1answer
21 views

Clustering by same random projection

I have $N$, $1024$-dimensional vectors. I want to cluster them by some similarity. Given the high dimensionality, standard metrics won't work. I tried a few Approximate Nearest Neighbor ...
1
vote
1answer
19 views

How to approach preprocessing large number features for machine learning?

I used to apply supervised machine learning for maximum few dozen "normal", natural features like human interpretable ones in Boston House Prices table. I usually try to understand each of them, think ...
0
votes
0answers
29 views

Comparing distances evaluated on different vector spaces

We have a dataset of I items who have been measured over two different sets of features A, with cardinality N, and B with cardinality M, and N > M. We would like to know in which feature space the ...
1
vote
0answers
52 views

MLE for high dimensional $\theta$

I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
2
votes
0answers
29 views

Texts on visualizing big data

I am looking for textbooks, papers or alternative material focusing on ideas and general principles for visualizing big data. By big data I primarily mean wide data, i.e. high-dimensional data, but I ...
0
votes
0answers
17 views

higher order dimensional understanding [duplicate]

In my application, 10-15 variables is common. It appears that I am missing some fundamentals maths associated with higher order. Can anyone suggest some basic papers or tutorials for me to get started?...
0
votes
0answers
25 views

visualising more than 3 dimensions [duplicate]

In my problem, I am having a lot of variables - easily 10-15 If this was a 2 dimensional data then I would plot these and visualize the plots to identify certain abnormalities. But with so many ...
1
vote
1answer
75 views

Autoencoders and/or PCA for highly sparse float vectors and a dataset of more than 2 million examples

I have a highly dimensional sparse dataset composed of 2.5 million of examples as follow : dataset_dimension=[2500000,360,280,18] Each example of this dataset ...
0
votes
0answers
9 views

($L_2$) distance for noisy data

I'm given a subspace $V$ and a set of $n$ corrupted observations $\tilde{x}_1 = x_1 +\epsilon_1,...,\tilde{x}_n = x_n + \epsilon_n \in \mathbb{R}^D$. Assume $D$ is large and that $\epsilon_i \sim N(0, ...
4
votes
0answers
29 views

Which approach based on the LASSO yields more biologically relevant results for gene data-sets?

I have a data-set with a continuous outcome variable and some confounding variables (like age, gender, ...) and many gene expressions (more than samples). The goal is to find relevant genes in ...
3
votes
1answer
209 views

Machine learning methods for multi-dimensional input and output

I have a large dataset where my input is an $M$-dimensional tensor, and each input has a corresponding $N$-dimensional output. My goal is to train a method to learn outputs from the millions of inputs ...
1
vote
1answer
40 views

How to sample a from p-variate discrete distribution when p is high

Suppose I have a $p$-dimensional random vector $X$, with probability mass function $f(x) = P(X = x)$, $x \in \mathbb \{0,1\}^p$. Each of the $p$ variables is binary. If I want to sample from the ...
0
votes
0answers
22 views

identifying multiD outliers

suppose you have two columns of data, one composed of numbers very close to 1 and numbers very close to 3, and the same for the ...
1
vote
1answer
24 views

Hierarchical Cluster Analysis of 100 objects with 114 variables each

I'm intending to make a cluster analysis of 100 objects. I've read a couple of books and determined that a Hierarchical agglomerative procedure with Ward's linkage method should be used in my case. As ...
1
vote
0answers
13 views

How does quantile regression behave when having more variables than observations?

When having a dataset with a larger number of variables than observations, it is not possible to build a linear regression model, because such a model would have an infinite number of possible ...
0
votes
0answers
7 views

Buffer needed on random vectors to have a desired distribution

I was wondering while reading about the curse of dimensionality, how many vectors in d dimensions, I would need to generate before I would be able to "accurately" deduce what distribution I have. ...
1
vote
1answer
50 views

Intrinsic dimensionality and density-based clustering

I’ve got several thousand observations in 350-dimensional space, in a relatively sparse matrix (median observation has 11 non-zero dimensions). I'm using a density-based clustering algorithm, DBSCAN, ...
1
vote
0answers
151 views

Can you use bray-curtis distances to evaluate standardized abundances in an NMDS?

So I'm trying to run an NMDS (using the vegan package in R) for a species X site matrix. Unfortunately the abundances of species are not inherently equal (If you can use that word) meaning that not ...
2
votes
0answers
56 views

Lower bounds on covering numbers for sparse vectors

Consider the set $S_k$, which is defined as the subset of $k$-sparse vectors in the unit sphere in $d$ dimensions: $$ S_k \triangleq \left\{ x \in \mathbb{R}^d : \| x \|_2 = 1, \, \left|\operatorname{...
0
votes
0answers
140 views

Best way to compare CNN outputs vectors

I was training a CNN (which contains only convolutional an pooling layers) to extract features of a given image. Output vector size is ...
0
votes
0answers
18 views

Modelling product purchase history as a random walk in n-space

I have a large dataset of customers making monthly purchases of multiple products. Customers usually purchase between 3 and 10 products, from a large product list (1000s). I'm interested in clustering ...
5
votes
1answer
86 views

Covering the unit sphere with sparse vectors

I'm looking for references for covering the $d$-dimensional unit sphere $$ \mathbb{S}^{d - 1} = \left\{ x \in \mathbb{R}^d : \| x \| = 1 \right\} $$ I'm trying to cover $\mathbb{S}^{d-1}$ with ...
1
vote
1answer
88 views

Curse of dimensionality introducing bias and variance

The Example presented in Elements of Statistical Learning: $Y = f(X) = e^{-8||X||^2}$. X is sampled between $[-1,1]$ is the underlying structure for Y. Authors use k- nearest neighbors. The argument ...
1
vote
1answer
18 views

Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
15
votes
1answer
346 views

High-dimensional regression: why is $\log p/n$ special?

I am trying to read up on the research in the area of high-dimensional regression; when $p$ is larger than $n$, that is, $p >> n$. It seems like the term $\log p/n$ appears often in terms of ...
0
votes
0answers
26 views

Is the survival given by a linear combination of a large number of iid normally distributed predictors stable? (simulation)

My question is about survival analysis, but I am quite sure that it may apply to regression in general. I will stick with a very simple simulation for survival times. For example, let's simulate the ...
1
vote
0answers
71 views

Graphs for Hyperparameter Exploration Results

I've recently conducted an extensive study into hyperparameter combinations for an LSTM. I varied the following hyperparameters: batch size number of neurons learning rate historical window size of ...
0
votes
0answers
98 views

How to create inliers inside a synthetic high dimensional dataset?

Since inliers have different definitions in different papers, the one I'm using is: "An inlier is a data value that lies in the interior of a statistical distribution and is in error." An example of ...