Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
1
vote
1answer
150 views

Autoencoders and/or PCA for highly sparse float vectors and a dataset of more than 2 million examples

I have a highly dimensional sparse dataset composed of 2.5 million of examples as follow : dataset_dimension=[2500000,360,280,18] Each example of this dataset ...
0
votes
0answers
23 views

Comparing dimensionality

I am reading a couple of papers, The first one claims that $p_n = O(n^a)$ for some $0 \leq a < \infty$ The second paper claims $p_n = o(\exp(n^\varepsilon))$ for some $0< \varepsilon < .2$ ...
4
votes
1answer
350 views

Kohonen SOM for high (50-100) dimensions

Does a Kohonen-style SOM, using Euclidean distance, work as well as, better than, or worse than alternatives (K-means, etc) in high (50-100 or more) dimensional space? EDIT: I'm thinking particularly ...
1
vote
1answer
153 views

Intrinsic dimensionality and density-based clustering

I’ve got several thousand observations in 350-dimensional space, in a relatively sparse matrix (median observation has 11 non-zero dimensions). I'm using a density-based clustering algorithm, DBSCAN, ...
3
votes
1answer
2k views

Advice for a sparse high-dimensional regression strategy

I have a regression problem where I would like to predict values given several thousand sparse features. The general data set is an $n \times m$ matrix where each row contains a sample with a value I ...
0
votes
1answer
195 views

How do you encode relationships between High-Cardinality, Categorical features?

I realize that some derivative of this question has been asked here before, but none have addressed the situation where there is ONLY high-cardinality, categorical data, and that the labels themselves ...
6
votes
2answers
159 views

Why is Gaussian distribution on high dimensional space like a soap bubble

In this famous post "Gaussian Distributions are Soap Bubbles" it is claimed that the distribution of the points looks like a soap bubble (where it is less dense in the center and more dense at the ...
0
votes
0answers
7 views

Appropriate machine learning technique for spectral data and low-frequency feedback

I have a performance measure and a data source that basically supplies a complex and varying waveform. I would like to apply some unstructured learning technique to try and find a pattern in the ...
6
votes
1answer
255 views

Expected squared distance from origin of training points vs. test points

This is from Exercise 2.4 (Page 39) of Elements of Statistical Learning: The edge effect problem discussed on page 23 is not peculiar to uniform sampling from bounded domains. Consider inputs drawn ...
0
votes
0answers
45 views

Can data ever be too high dimensional for the Lasso?

I'm trying to implement Lasso on high dimensional textual data. Format of Data: p ~= 45,000, n~=4,000 When running the Lasso, I get a training score of 0 and the number of features selected as 0. ...
12
votes
1answer
4k views

Does Dimensionality curse effect some models more than others?

The places I have been reading about dimensionality curse explain it in conjunction to kNN primarily, and linear models in general. I regularly see top rankers in Kaggle using thousands of features on ...
0
votes
0answers
23 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
1
vote
1answer
172 views

What are some good resources to learn Statistical Genetics?

I have a B.S. and M.S. degree in Statistics. I have experience in R. I know the basic structures of Python [Expertise Level: Beginner]. I would really appreciate if you can share with me some ...
1
vote
2answers
92 views

Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...
2
votes
1answer
59 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
0
votes
1answer
23 views

Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0
votes
0answers
22 views

Neural network model has way more features than samples but yields good test accuracy

I am recently doing a bioinformatic machine learning project. We have over 470,000 features and only 700 training samples and 300 test samples. We used a perceptron with 1 hidden layer to train. ...
2
votes
1answer
284 views

Unsupervised Learning on Multilevel/Multidimensional Data

I am working on a case-control study, where I for each individual have high dimensional data (like illustrated in the image). I would like to do both PCA analysis and Clustering on this data, but it ...
4
votes
1answer
1k views

How does glmnet handle larger datasets?

I'm looking to fit a model with about 1k-40k variables and up to a few million observations. Can anyone with a bit more experience speak to its performance for larger datasets? It looks like I can ...
2
votes
0answers
280 views

How to do Hierarchical (Nested) Elliptical Copula simulation sampling

I am doing a project to aggregate about 30 risks into total loss (15 of them are market risks, and 15 of them are insurance risks). The current approach is to simulate millions of scenario with ...
4
votes
2answers
5k views

How does linear SVMs function in multi dimensional feature space?

How does linear SVMs function in multi dimensional feature space? I'm not able to picture how a linear SVM can perform classification in more than 2 dimensions. Also, when to chose linear SVMs and ...
0
votes
2answers
38 views

Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
3
votes
1answer
215 views

Multi Categorical Features vs multiple Features for categories

Say I am discretizing continuous data based on percentiles. (I realize this is generally frowned upon, but I am doing this for the sake of experiment) I am trying different percentiles, eg breaking ...
1
vote
1answer
36 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
0answers
32 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
0
votes
2answers
30 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
54
votes
7answers
26k views

Best PCA algorithm for huge number of features (>10K)?

I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO. It's kind of at the intersection between statistics and ...
1
vote
0answers
21 views

How to approach the calculations of probabilities in high dimensions?

I don't really have too much trouble finding probabilities using joint probability density functions (PDFs) (of two variables) by drawing the area of support in the $xy$-plane, and then integrating ...
236
votes
8answers
70k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
0
votes
0answers
15 views

Dimensionality of similarity matrix

Below is a screen shot of a paper. The authors take a data-set $E\in R^{nxm}$. Here $n$ is the number of observations/samples/patients and $m$ is the number of genes/features. Preprocessing ...
4
votes
1answer
38 views

Which approach based on the LASSO yields more biologically relevant results for gene data-sets?

I have a data-set with a continuous outcome variable and some confounding variables (like age, gender, ...) and many gene expressions (more than samples). The goal is to find relevant genes in ...
0
votes
0answers
10 views

What kind of algorithms are appropriate for this sort of medium-dimensional integration problem?

I'm trying to model a situation in which an agent must select one of several choices (not more than ten). Each choice is associated with a vector, known to the agent, representing its effectiveness in ...
1
vote
1answer
52 views

The largest dimension of feature spaces that the logistic regression can handle?

The estimation in the logistic regression (https://retostauffer.github.io/Rfoehnix/articles/logisticregression.html) is via the Newton method where the computed Hessian is given as $$ H = -X^TWX $$ ...
5
votes
2answers
188 views

Align noisy point clouds

I have a point cloud $X$ that, I suspect, is a translate of a Gaussian-corrupted version of a subset of a larger cloud $Y$, both high-dimensional ($d$ is at least 100 and ideally 10,000). What is the ...
0
votes
0answers
120 views

Does High Dimensional Data effects SVM?

As we move into higher dimensions, we will find even more corners. This will make an ever increasing percentage of the total space available. Now imagine we have data spread across some ...
0
votes
0answers
25 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
0
votes
0answers
88 views

Variable selection in high dimensionality

I was wondering what are some techniques for variable selection when there are a large number of variables lets say 1000, and the entire dataset is too large to fit into memory. How would one go ...
2
votes
1answer
121 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
0answers
25 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
9
votes
2answers
285 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
1answer
25 views

Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0
votes
0answers
31 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
0
votes
0answers
19 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
1
vote
0answers
78 views

e-SVM performance vs number of feature

I apply epsilon Support Vector Machine (e-SVM) to a regression problem via Weka. I have about 6000 features and 2000 samples. I order the feature respect to minimal-redundancy-maximal-relevance ...
0
votes
0answers
6 views

How to combine exposure measurements with a job exposure matrix

In order to better estimate occupational exposure to chemicals in the general worker population, I'd like to combine a job exposure matrix (JEM) with chemical exposure measurements. A generic JEM is ...
0
votes
0answers
14 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
2answers
333 views

FDR and Bonferroni corrections and Logistic Regression / Classification in High Dimensional Space

My work involves Classification --e.g. Logistic Regressions-- in a relatively High Dimensional setting (i.e. 300 to 1,500 variables). I wonder if the Bonferroni and FDR corrections have any relevance ...
0
votes
0answers
13 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
1answer
37 views

Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
0
votes
0answers
27 views

Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...