# Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

244 questions
Filter by
Sorted by
Tagged with
1answer
150 views

### Autoencoders and/or PCA for highly sparse float vectors and a dataset of more than 2 million examples

I have a highly dimensional sparse dataset composed of 2.5 million of examples as follow : dataset_dimension=[2500000,360,280,18] Each example of this dataset ...
0answers
23 views

### Comparing dimensionality

I am reading a couple of papers, The first one claims that $p_n = O(n^a)$ for some $0 \leq a < \infty$ The second paper claims $p_n = o(\exp(n^\varepsilon))$ for some $0< \varepsilon < .2$ ...
1answer
350 views

### Kohonen SOM for high (50-100) dimensions

Does a Kohonen-style SOM, using Euclidean distance, work as well as, better than, or worse than alternatives (K-means, etc) in high (50-100 or more) dimensional space? EDIT: I'm thinking particularly ...
1answer
153 views

### Intrinsic dimensionality and density-based clustering

I’ve got several thousand observations in 350-dimensional space, in a relatively sparse matrix (median observation has 11 non-zero dimensions). I'm using a density-based clustering algorithm, DBSCAN, ...
1answer
2k views

### Advice for a sparse high-dimensional regression strategy

I have a regression problem where I would like to predict values given several thousand sparse features. The general data set is an $n \times m$ matrix where each row contains a sample with a value I ...
1answer
195 views

### How do you encode relationships between High-Cardinality, Categorical features?

I realize that some derivative of this question has been asked here before, but none have addressed the situation where there is ONLY high-cardinality, categorical data, and that the labels themselves ...
2answers
159 views

### Why is Gaussian distribution on high dimensional space like a soap bubble

In this famous post "Gaussian Distributions are Soap Bubbles" it is claimed that the distribution of the points looks like a soap bubble (where it is less dense in the center and more dense at the ...
0answers
7 views

### Appropriate machine learning technique for spectral data and low-frequency feedback

I have a performance measure and a data source that basically supplies a complex and varying waveform. I would like to apply some unstructured learning technique to try and find a pattern in the ...
1answer
255 views

### Expected squared distance from origin of training points vs. test points

This is from Exercise 2.4 (Page 39) of Elements of Statistical Learning: The edge effect problem discussed on page 23 is not peculiar to uniform sampling from bounded domains. Consider inputs drawn ...
0answers
45 views

### Can data ever be too high dimensional for the Lasso?

I'm trying to implement Lasso on high dimensional textual data. Format of Data: p ~= 45,000, n~=4,000 When running the Lasso, I get a training score of 0 and the number of features selected as 0. ...
1answer
4k views

### Does Dimensionality curse effect some models more than others?

The places I have been reading about dimensionality curse explain it in conjunction to kNN primarily, and linear models in general. I regularly see top rankers in Kaggle using thousands of features on ...
0answers
23 views

### Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
1answer
172 views

### What are some good resources to learn Statistical Genetics?

I have a B.S. and M.S. degree in Statistics. I have experience in R. I know the basic structures of Python [Expertise Level: Beginner]. I would really appreciate if you can share with me some ...
2answers
92 views

### Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...
1answer
59 views

### Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
1answer
23 views

### Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0answers
22 views

### Neural network model has way more features than samples but yields good test accuracy

I am recently doing a bioinformatic machine learning project. We have over 470,000 features and only 700 training samples and 300 test samples. We used a perceptron with 1 hidden layer to train. ...
1answer
284 views

### Unsupervised Learning on Multilevel/Multidimensional Data

I am working on a case-control study, where I for each individual have high dimensional data (like illustrated in the image). I would like to do both PCA analysis and Clustering on this data, but it ...
1answer
1k views

### How does glmnet handle larger datasets?

I'm looking to fit a model with about 1k-40k variables and up to a few million observations. Can anyone with a bit more experience speak to its performance for larger datasets? It looks like I can ...
0answers
280 views

### How to do Hierarchical (Nested) Elliptical Copula simulation sampling

I am doing a project to aggregate about 30 risks into total loss (15 of them are market risks, and 15 of them are insurance risks). The current approach is to simulate millions of scenario with ...
2answers
5k views

### How does linear SVMs function in multi dimensional feature space?

How does linear SVMs function in multi dimensional feature space? I'm not able to picture how a linear SVM can perform classification in more than 2 dimensions. Also, when to chose linear SVMs and ...
2answers
38 views

### Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
1answer
215 views

### Multi Categorical Features vs multiple Features for categories

Say I am discretizing continuous data based on percentiles. (I realize this is generally frowned upon, but I am doing this for the sake of experiment) I am trying different percentiles, eg breaking ...
1answer
36 views

### Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0answers
32 views

2answers
188 views

### Align noisy point clouds

I have a point cloud $X$ that, I suspect, is a translate of a Gaussian-corrupted version of a subset of a larger cloud $Y$, both high-dimensional ($d$ is at least 100 and ideally 10,000). What is the ...
0answers
120 views

### Does High Dimensional Data effects SVM?

As we move into higher dimensions, we will find even more corners. This will make an ever increasing percentage of the total space available. Now imagine we have data spread across some ...
0answers
25 views

### estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
0answers
88 views

### Variable selection in high dimensionality

I was wondering what are some techniques for variable selection when there are a large number of variables lets say 1000, and the entire dataset is too large to fit into memory. How would one go ...
1answer
121 views

### estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0answers
25 views

### Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
2answers
285 views

### Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
1answer
25 views

### Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0answers
31 views

### Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
0answers
19 views

### Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
0answers
78 views

### e-SVM performance vs number of feature

I apply epsilon Support Vector Machine (e-SVM) to a regression problem via Weka. I have about 6000 features and 2000 samples. I order the feature respect to minimal-redundancy-maximal-relevance ...
0answers
6 views

### How to combine exposure measurements with a job exposure matrix

In order to better estimate occupational exposure to chemicals in the general worker population, I'd like to combine a job exposure matrix (JEM) with chemical exposure measurements. A generic JEM is ...
0answers
14 views

### Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
2answers
333 views

### FDR and Bonferroni corrections and Logistic Regression / Classification in High Dimensional Space

My work involves Classification --e.g. Logistic Regressions-- in a relatively High Dimensional setting (i.e. 300 to 1,500 variables). I wonder if the Bonferroni and FDR corrections have any relevance ...
0answers
13 views

### Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
1answer
37 views

### Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
0answers
27 views

### Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...