Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [high-dimensional]

Pertains to great number of features or many dimensions (variables) for data points. (For great number of data points, use the tag 'large-data'.)

1
vote
1answer
67 views
+50

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
0answers
10 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
0
votes
1answer
23 views

Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0
votes
0answers
12 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
8
votes
2answers
193 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
32 views

Pearson correlation as distance metric

I have read that Pearson correlation is used as a similarity measure for high dimensional data like genetic data. Could someone please explain it intuitively, mathematically with reference as to why ...
0
votes
0answers
11 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
0
votes
0answers
13 views

metrics or distributions in high-dimensional spaces?

This is a somewhat naive question that occurred to me while thinking about organisms that potentially have to deal with high-dimensional sensory input. I wonder which problem is harder in theory: ...
0
votes
0answers
6 views

How to combine exposure measurements with a job exposure matrix

In order to better estimate occupational exposure to chemicals in the general worker population, I'd like to combine a job exposure matrix (JEM) with chemical exposure measurements. A generic JEM is ...
0
votes
0answers
5 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
10 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
1answer
17 views

Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
0
votes
0answers
10 views

Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...
2
votes
1answer
28 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
12 views

is random projection a linear or non-linear feature extraction method?

The dimensionality reduction has two different types: feature selection and feature selection. As far as i know, the random projection cannot be a feature selection method. Therefore, is it a linear ...
0
votes
0answers
34 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
1
vote
1answer
34 views

State space with lasso

Is it possible to incorporate lasso variable selection in the high dimensional state space model. If yes, is there any code or package available in R
0
votes
1answer
22 views

Euclidean distances in spaces of different dimensions

A modest attempt to illustrate my question: Setup Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, ...
0
votes
1answer
32 views

Application of LASSO , Ridge, PLS in feature selection of spectral data

The meatspec data in faraway package is spectral data with 100 features .(215 *101). Use of LASSO over ridge and PLS gives better performance (RMSE based) But none of the features are removed ( no ...
0
votes
0answers
15 views

Implementation of Compressed Regression (examples?)

I found several articles discussing compressed regression, whether in the bayesian framework or ordinary LS/GLM (1, 2, 3 and others). The idea seems simple, the model becomes $$ Y = \Phi X \beta + \...
0
votes
0answers
35 views

Main differences (problems and mathematics) between traditional statistics and high dimensional statistics

High dimensional statistics seems to be hot nowadays. What are the main differences, in terms of questions and problems it tries to solve, as well as the mathematical tools used, between "traditional"...
1
vote
2answers
31 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
1
vote
0answers
43 views

t-SNE for finding nearest neighbors

I had a question about dimensionality reduction for finding nearest neighbors and was hoping someone could help me out here. Suppose I have good features for images, say penultimate layer features ...
9
votes
1answer
170 views

How do children manage to pull their parents together in a PCA projection of a GWAS data set?

Take 20 random points in a 10,000-dimensional space with each coordinate iid from $\mathcal N(0,1)$. Split them into 10 pairs ("couples") and add the average of each pair ("a child") to the dataset. ...
1
vote
0answers
15 views

margin point of view of logistic regression

I am currently studying Lasso for logistic regression and am using Buhlmann et.al book (Statistics for High Dimensional Data) to understand better. Section In this book on page 3.3.1 they define ...
3
votes
0answers
104 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
0
votes
1answer
25 views

Classification problem for continous target

I have a big dataset with around 100k samples and 2k real-value features. The target variable is in [-1,1], but its distribution is highly concentrated around zero (...
1
vote
1answer
122 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
0
votes
0answers
12 views

non-normal censored outcome, high dimensional predictors

I have an outcome that is non-normal (multimodal) and has an upper limit of detection giving censored values at an upper bound. I have several hundred observations and wish to predict these non-normal ...
0
votes
0answers
13 views

Are there any plots for the results of the Lasso estimator besides plotting the Lasso path?

When one reports the results of methods like Lasso, group Lasso or Stability Selection, are there any nice plots one could generate for genome-wide association studies (besides lasso paths) to make ...
2
votes
0answers
28 views

PCA: mean of marginal distribution of high-dimensional vector

Consider the following probabilistic model: $$p(x) = \mathcal{N}(0, I_d), \ x \in \mathbb{R}^d$$ $$p(y|x) = \mathcal{N}(Wx + \mu, \sigma^2I_D), \ y, \mu \in \mathbb{R}^D, W \in \mathbb{R}^{D\times d}...
1
vote
0answers
43 views

algorithms for high dimensional data in a small sample size

I am trying to analysis high-dimensional data in a small sample size and I meet a problem: First,I divided all samples into two sets: training set and test set. Second, in the training set The Y is ...
0
votes
1answer
22 views

Clustering by same random projection

I have $N$, $1024$-dimensional vectors. I want to cluster them by some similarity. Given the high dimensionality, standard metrics won't work. I tried a few Approximate Nearest Neighbor ...
1
vote
1answer
21 views

How to approach preprocessing large number features for machine learning?

I used to apply supervised machine learning for maximum few dozen "normal", natural features like human interpretable ones in Boston House Prices table. I usually try to understand each of them, think ...
0
votes
0answers
32 views

Comparing distances evaluated on different vector spaces

We have a dataset of I items who have been measured over two different sets of features A, with cardinality N, and B with cardinality M, and N > M. We would like to know in which feature space the ...
1
vote
0answers
69 views

MLE for high dimensional $\theta$

I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
2
votes
0answers
30 views

Texts on visualizing big data

I am looking for textbooks, papers or alternative material focusing on ideas and general principles for visualizing big data. By big data I primarily mean wide data, i.e. high-dimensional data, but I ...
0
votes
0answers
17 views

higher order dimensional understanding [duplicate]

In my application, 10-15 variables is common. It appears that I am missing some fundamentals maths associated with higher order. Can anyone suggest some basic papers or tutorials for me to get started?...
0
votes
0answers
25 views

visualising more than 3 dimensions [duplicate]

In my problem, I am having a lot of variables - easily 10-15 If this was a 2 dimensional data then I would plot these and visualize the plots to identify certain abnormalities. But with so many ...
1
vote
1answer
103 views

Autoencoders and/or PCA for highly sparse float vectors and a dataset of more than 2 million examples

I have a highly dimensional sparse dataset composed of 2.5 million of examples as follow : dataset_dimension=[2500000,360,280,18] Each example of this dataset ...
0
votes
0answers
10 views

($L_2$) distance for noisy data

I'm given a subspace $V$ and a set of $n$ corrupted observations $\tilde{x}_1 = x_1 +\epsilon_1,...,\tilde{x}_n = x_n + \epsilon_n \in \mathbb{R}^D$. Assume $D$ is large and that $\epsilon_i \sim N(0, ...
4
votes
0answers
29 views

Which approach based on the LASSO yields more biologically relevant results for gene data-sets?

I have a data-set with a continuous outcome variable and some confounding variables (like age, gender, ...) and many gene expressions (more than samples). The goal is to find relevant genes in ...
3
votes
1answer
380 views

Machine learning methods for multi-dimensional input and output

I have a large dataset where my input is an $M$-dimensional tensor, and each input has a corresponding $N$-dimensional output. My goal is to train a method to learn outputs from the millions of inputs ...
1
vote
1answer
40 views

How to sample a from p-variate discrete distribution when p is high

Suppose I have a $p$-dimensional random vector $X$, with probability mass function $f(x) = P(X = x)$, $x \in \mathbb \{0,1\}^p$. Each of the $p$ variables is binary. If I want to sample from the ...
0
votes
0answers
23 views

identifying multiD outliers

suppose you have two columns of data, one composed of numbers very close to 1 and numbers very close to 3, and the same for the ...
1
vote
1answer
24 views

Hierarchical Cluster Analysis of 100 objects with 114 variables each

I'm intending to make a cluster analysis of 100 objects. I've read a couple of books and determined that a Hierarchical agglomerative procedure with Ward's linkage method should be used in my case. As ...
1
vote
0answers
16 views

How does quantile regression behave when having more variables than observations?

When having a dataset with a larger number of variables than observations, it is not possible to build a linear regression model, because such a model would have an infinite number of possible ...
0
votes
0answers
7 views

Buffer needed on random vectors to have a desired distribution

I was wondering while reading about the curse of dimensionality, how many vectors in d dimensions, I would need to generate before I would be able to "accurately" deduce what distribution I have. ...
1
vote
1answer
84 views

Intrinsic dimensionality and density-based clustering

I’ve got several thousand observations in 350-dimensional space, in a relatively sparse matrix (median observation has 11 non-zero dimensions). I'm using a density-based clustering algorithm, DBSCAN, ...
1
vote
0answers
193 views

Can you use bray-curtis distances to evaluate standardized abundances in an NMDS?

So I'm trying to run an NMDS (using the vegan package in R) for a species X site matrix. Unfortunately the abundances of species are not inherently equal (If you can use that word) meaning that not ...