Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

how does noise-to-signal ratio effect the data splitting in training,validation and test sets

How will we divide our data set in training, validation and test set for model selection and model assessment? As in the book "Elements of statistical learning" page no.222 the author has mentioned ...
0
votes
1answer
21 views

How to cut features with large amount of 0 values from high dimensional data?

I have genomic data (miRNA) that is high dimensional: $198$ samples and $1584$ features. ...
0
votes
0answers
14 views

Unsupervised Clustering high dimentional data not having estimation for K

I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster ...
3
votes
1answer
83 views

Difference between big data and high dimensional data

What do big data and high dimensional data mean? Is high dimensional data a special case of big data? What are the complications that arise in the analysis of high dimensional and big data each?
2
votes
0answers
26 views

How can I do subject clustering in a high dimension?

I am working on a dataset with 200 participants' data. For each participant, we collected 20-70 days' data. After data engineering, we got features over 300 dimensions. Do we have any way to cluster ...
0
votes
0answers
24 views

Comparing dimensionality

I am reading a couple of papers, The first one claims that $p_n = O(n^a)$ for some $0 \leq a < \infty$ The second paper claims $p_n = o(\exp(n^\varepsilon))$ for some $0< \varepsilon < .2$ ...
6
votes
2answers
215 views

Why is Gaussian distribution on high dimensional space like a soap bubble

In this famous post "Gaussian Distributions are Soap Bubbles" it is claimed that the distribution of the points looks like a soap bubble (where it is less dense in the center and more dense at the ...
0
votes
0answers
45 views

Can data ever be too high dimensional for the Lasso?

I'm trying to implement Lasso on high dimensional textual data. Format of Data: p ~= 45,000, n~=4,000 When running the Lasso, I get a training score of 0 and the number of features selected as 0. ...
3
votes
1answer
61 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
1
vote
2answers
127 views

Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...
0
votes
1answer
46 views

Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0
votes
0answers
8 views

Appropriate machine learning technique for spectral data and low-frequency feedback

I have a performance measure and a data source that basically supplies a complex and varying waveform. I would like to apply some unstructured learning technique to try and find a pattern in the ...
0
votes
0answers
24 views

Neural network model has way more features than samples but yields good test accuracy

I am recently doing a bioinformatic machine learning project. We have over 470,000 features and only 700 training samples and 300 test samples. We used a perceptron with 1 hidden layer to train. ...
2
votes
1answer
80 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
0
votes
2answers
40 views

Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
1
vote
1answer
51 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
0answers
36 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
0
votes
2answers
31 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
1
vote
0answers
22 views

How to approach the calculations of probabilities in high dimensions?

I don't really have too much trouble finding probabilities using joint probability density functions (PDFs) (of two variables) by drawing the area of support in the $xy$-plane, and then integrating ...
0
votes
0answers
15 views

Dimensionality of similarity matrix

Below is a screen shot of a paper. The authors take a data-set $E\in R^{nxm}$. Here $n$ is the number of observations/samples/patients and $m$ is the number of genes/features. Preprocessing ...
0
votes
0answers
11 views

What kind of algorithms are appropriate for this sort of medium-dimensional integration problem?

I'm trying to model a situation in which an agent must select one of several choices (not more than ten). Each choice is associated with a vector, known to the agent, representing its effectiveness in ...
1
vote
1answer
53 views

The largest dimension of feature spaces that the logistic regression can handle?

The estimation in the logistic regression (https://retostauffer.github.io/Rfoehnix/articles/logisticregression.html) is via the Newton method where the computed Hessian is given as $$ H = -X^TWX $$ ...
0
votes
0answers
144 views

Does High Dimensional Data effects SVM?

As we move into higher dimensions, we will find even more corners. This will make an ever increasing percentage of the total space available. Now imagine we have data spread across some ...
0
votes
0answers
29 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
3
votes
1answer
128 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
0answers
33 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
0
votes
1answer
26 views

Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0
votes
0answers
40 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
9
votes
2answers
290 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
23 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
0
votes
0answers
6 views

How to combine exposure measurements with a job exposure matrix

In order to better estimate occupational exposure to chemicals in the general worker population, I'd like to combine a job exposure matrix (JEM) with chemical exposure measurements. A generic JEM is ...
0
votes
0answers
17 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
13 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
1answer
40 views

Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
0
votes
0answers
32 views

Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...
3
votes
1answer
47 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
20 views

is random projection a linear or non-linear feature extraction method?

The dimensionality reduction has two different types: feature selection and feature selection. As far as i know, the random projection cannot be a feature selection method. Therefore, is it a linear ...
0
votes
0answers
49 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
1
vote
1answer
72 views

State space with lasso

Is it possible to incorporate lasso variable selection in the high dimensional state space model. If yes, is there any code or package available in R
0
votes
1answer
38 views

Euclidean distances in spaces of different dimensions

A modest attempt to illustrate my question: Setup Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, ...
0
votes
1answer
74 views

Application of LASSO , Ridge, PLS in feature selection of spectral data

The meatspec data in faraway package is spectral data with 100 features .(215 *101). Use of LASSO over ridge and PLS gives better performance (RMSE based) But none of the features are removed ( no ...
0
votes
0answers
17 views

Implementation of Compressed Regression (examples?)

I found several articles discussing compressed regression, whether in the bayesian framework or ordinary LS/GLM (1, 2, 3 and others). The idea seems simple, the model becomes $$ Y = \Phi X \beta + \...
0
votes
0answers
37 views

Main differences (problems and mathematics) between traditional statistics and high dimensional statistics

High dimensional statistics seems to be hot nowadays. What are the main differences, in terms of questions and problems it tries to solve, as well as the mathematical tools used, between "traditional"...
1
vote
2answers
87 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
1
vote
0answers
157 views

t-SNE for finding nearest neighbors

I had a question about dimensionality reduction for finding nearest neighbors and was hoping someone could help me out here. Suppose I have good features for images, say penultimate layer features ...
9
votes
1answer
194 views

How do children manage to pull their parents together in a PCA projection of a GWAS data set?

Take 20 random points in a 10,000-dimensional space with each coordinate iid from $\mathcal N(0,1)$. Split them into 10 pairs ("couples") and add the average of each pair ("a child") to the dataset. ...
1
vote
0answers
21 views

margin point of view of logistic regression

I am currently studying Lasso for logistic regression and am using Buhlmann et.al book (Statistics for High Dimensional Data) to understand better. Section In this book on page 3.3.1 they define ...
3
votes
0answers
240 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
1
vote
1answer
28 views

Classification problem for continous target

I have a big dataset with around 100k samples and 2k real-value features. The target variable is in [-1,1], but its distribution is highly concentrated around zero (...
1
vote
1answer
183 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...