Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Visualization of high dimensional data on grid

I am interested in some analysis of a high dimensional domain (d=32). In a 2 dimensional domain, I construct a grid, evaluate given function over the domain and either scatter plot the points or plot ...
0
votes
1answer
21 views

Does a high-dimensionality of a time series refer to its length, number of variables or both?

I'd like to cluster some time series that describe a flow of a variable (say, temperature) throughout a day. Measurements are made every 5 minutes so each time series has 288 values. Are we talking ...
1
vote
0answers
35 views

Sparse PCA for $p >> n$ solution with Elastic Net

I was reading about the sparse principal component approach by Zou, Hastie and Tibshirani but I do not quite understand how they handle the $p \gg n$ case in their paper. To derive the sparse axis, ...
1
vote
0answers
16 views

Visualise high dimensional data - returns of n-asset portfolio as 2D plot, like heatmap?

It's possible to visualise both asset allocations and returns of 2-assets portfolio as 2D heatmap. Like in image below, the visualisation of $[Gold, Silver]$ portfolio with restriction $Gold + ...
0
votes
0answers
37 views

Mean shift clustering and the curse of dimensionality

I've often come across resources that mention that mean shift based clustering doesn't work well in higher dimensions. The sources are as follows: Page 1 of https://www.ncbi.nlm.nih.gov/pmc/articles/...
0
votes
0answers
33 views

Can we always perform SVD on the data matrix before doing high-dimensional logistic regression?

So I'm using lasso logistic regression to classify my data. My data matrix $X$ has dimension $n\times p$ for $p >> n$. As $p$ is on the order of a billion, I expect to face some computational ...
0
votes
1answer
46 views

using finite difference to estimate high dimensional gradient in gradient descent methods

I'm not very familiar with optimization problem, but I know that if the gradient of function is hard to find, it can use finite difference method to estimate it. Like scipy.minimize, it would use this ...
0
votes
0answers
8 views

Visualization and two-group statistical testing in high-dimension binary data

I am planning a biomarker discovery experiment, and the data is expected to take the form of a high-dimensional dataset where each data point is described by 8 binary variables indicating the presence ...
0
votes
0answers
8 views

In statistics, do variable screening and variable selection have same meaning?

I thought they had the same meaning.But some paper related to screening would propose new screening method,then it says ...
0
votes
0answers
26 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
1
vote
0answers
34 views

Pyro Gaussian processes regression achieves a low error, but then always outputs the same prediction on new data

I have tried experimenting with high dimensional Gaussian processes multiple times and I always get the same result. The model trains, and then when it comes time to predict with some new data (or ...
0
votes
0answers
10 views

Does reducing dimensionality of data makes it less linearly separable?

I recently read about kernel trick in SVM that says that mapping data to higher dimensions makes it more linearly separable but can we conversely say that "mapping ...
1
vote
0answers
26 views

Regression on unevenly distributed high dimensional dataset

I have a very high dimensional (20K+ hand engineered features) biological dataset to predict a single continuous output variable (such as a mental state exam scores for a dementia patient). The output ...
0
votes
0answers
7 views

method for determing the important factor for high dimension categorical data

I have around 1000 people with total 400 categorical features, but each one will only have subset of those 400 features(ranging from 3-60 for this population), thus the dataset is fairly sparse. Now I ...
1
vote
1answer
34 views

how does noise-to-signal ratio effect the data splitting in training,validation and test sets

How will we divide our data set in training, validation and test set for model selection and model assessment? As in the book "Elements of statistical learning" page no.222 the author has mentioned ...
0
votes
1answer
24 views

How to cut features with large amount of 0 values from high dimensional data?

I have genomic data (miRNA) that is high dimensional: $198$ samples and $1584$ features. ...
0
votes
0answers
15 views

Unsupervised Clustering high dimentional data not having estimation for K

I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster ...
3
votes
1answer
299 views

Difference between big data and high dimensional data

What do big data and high dimensional data mean? Is high dimensional data a special case of big data? What are the complications that arise in the analysis of high dimensional and big data each?
2
votes
0answers
32 views

How can I do subject clustering in a high dimension?

I am working on a dataset with 200 participants' data. For each participant, we collected 20-70 days' data. After data engineering, we got features over 300 dimensions. Do we have any way to cluster ...
0
votes
0answers
27 views

Comparing dimensionality

I am reading a couple of papers, The first one claims that $p_n = O(n^a)$ for some $0 \leq a < \infty$ The second paper claims $p_n = o(\exp(n^\varepsilon))$ for some $0< \varepsilon < .2$ ...
8
votes
2answers
431 views

Why is Gaussian distribution on high dimensional space like a soap bubble

In this famous post "Gaussian Distributions are Soap Bubbles" it is claimed that the distribution of the points looks like a soap bubble (where it is less dense in the center and more dense at the ...
0
votes
0answers
51 views

Can data ever be too high dimensional for the Lasso?

I'm trying to implement Lasso on high dimensional textual data. Format of Data: p ~= 45,000, n~=4,000 When running the Lasso, I get a training score of 0 and the number of features selected as 0. ...
3
votes
1answer
111 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
1
vote
2answers
224 views

Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...
0
votes
1answer
117 views

Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0
votes
0answers
8 views

Appropriate machine learning technique for spectral data and low-frequency feedback

I have a performance measure and a data source that basically supplies a complex and varying waveform. I would like to apply some unstructured learning technique to try and find a pattern in the ...
0
votes
0answers
42 views

Neural network model has way more features than samples but yields good test accuracy

I am recently doing a bioinformatic machine learning project. We have over 470,000 features and only 700 training samples and 300 test samples. We used a perceptron with 1 hidden layer to train. ...
2
votes
1answer
177 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
0
votes
2answers
58 views

Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
2
votes
1answer
146 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
0answers
43 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
0
votes
2answers
41 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
1
vote
0answers
23 views

How to approach the calculations of probabilities in high dimensions?

I don't really have too much trouble finding probabilities using joint probability density functions (PDFs) (of two variables) by drawing the area of support in the $xy$-plane, and then integrating ...
0
votes
0answers
16 views

Dimensionality of similarity matrix

Below is a screen shot of a paper. The authors take a data-set $E\in R^{nxm}$. Here $n$ is the number of observations/samples/patients and $m$ is the number of genes/features. Preprocessing ...
0
votes
0answers
11 views

What kind of algorithms are appropriate for this sort of medium-dimensional integration problem?

I'm trying to model a situation in which an agent must select one of several choices (not more than ten). Each choice is associated with a vector, known to the agent, representing its effectiveness in ...
1
vote
1answer
55 views

The largest dimension of feature spaces that the logistic regression can handle?

The estimation in the logistic regression (https://retostauffer.github.io/Rfoehnix/articles/logisticregression.html) is via the Newton method where the computed Hessian is given as $$ H = -X^TWX $$ ...
0
votes
0answers
155 views

Does High Dimensional Data effects SVM?

As we move into higher dimensions, we will find even more corners. This will make an ever increasing percentage of the total space available. Now imagine we have data spread across some ...
0
votes
0answers
33 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
3
votes
1answer
154 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
0answers
66 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
0
votes
1answer
26 views

Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0
votes
0answers
54 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
9
votes
2answers
305 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
27 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
0
votes
0answers
13 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
1answer
52 views

Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
3
votes
1answer
56 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
58 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
1
vote
1answer
84 views

State space with lasso

Is it possible to incorporate lasso variable selection in the high dimensional state space model. If yes, is there any code or package available in R
0
votes
1answer
61 views

Euclidean distances in spaces of different dimensions

A modest attempt to illustrate my question: Setup Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, ...

1 2 3 4 5 6