Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
0
votes
0answers
37 views

Can data ever be too high dimensional for the Lasso?

I'm trying to implement Lasso on high dimensional textual data. Format of Data: p ~= 45,000, n~=4,000 When running the Lasso, I get a training score of 0 and the number of features selected as 0. ...
0
votes
0answers
20 views

Why isn't a gaussian kernel subject to the curse of dimensionality?

This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(...
1
vote
2answers
55 views

Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...
0
votes
1answer
19 views

Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0
votes
0answers
5 views

Appropriate machine learning technique for spectral data and low-frequency feedback

I hope this naive question is appropriate here -- my statistical understanding is mainly econometrics, but I have a (non-commercial) research opportunity that requires a more machine-learning approach,...
0
votes
0answers
21 views

Neural network model has way more features than samples but yields good test accuracy

I am recently doing a bioinformatic machine learning project. We have over 470,000 features and only 700 training samples and 300 test samples. We used a perceptron with 1 hidden layer to train. ...
2
votes
1answer
46 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
0
votes
2answers
32 views

Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
1
vote
1answer
30 views

Clustering data with covariance for each point

I am looking to cluster data points that each have a covariance around itself (based on some function of its neighbourhood, but how I got it is not important). I would like to use the covariance to ...
0
votes
0answers
28 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
0
votes
2answers
29 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
1
vote
0answers
21 views

How to approach the calculations of probabilities in high dimensions?

I don't really have too much trouble finding probabilities using joint probability density functions (PDFs) (of two variables) by drawing the area of support in the $xy$-plane, and then integrating ...
0
votes
0answers
14 views

Dimensionality of similarity matrix

Below is a screen shot of a paper. The authors take a data-set $E\in R^{nxm}$. Here $n$ is the number of observations/samples/patients and $m$ is the number of genes/features. Preprocessing ...
0
votes
0answers
9 views

What kind of algorithms are appropriate for this sort of medium-dimensional integration problem?

I'm trying to model a situation in which an agent must select one of several choices (not more than ten). Each choice is associated with a vector, known to the agent, representing its effectiveness in ...
1
vote
1answer
52 views

The largest dimension of feature spaces that the logistic regression can handle?

The estimation in the logistic regression (https://retostauffer.github.io/Rfoehnix/articles/logisticregression.html) is via the Newton method where the computed Hessian is given as $$ H = -X^TWX $$ ...
0
votes
0answers
93 views

Does High Dimensional Data effects SVM?

As we move into higher dimensions, we will find even more corners. This will make an ever increasing percentage of the total space available. Now imagine we have data spread across some ...
0
votes
0answers
22 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
2
votes
1answer
117 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
0answers
23 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
0
votes
1answer
25 views

Correct approach to testing for homogeneity of variance across ~2000 conditional distributions?

I'm not well-versed in statistics so apologies in advance for struggling to ask this question the right way. Essentially, I have 1836 timeseries of stock prices. For each of these timeseries, I am ...
0
votes
0answers
25 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
9
votes
2answers
281 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
18 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
0
votes
0answers
6 views

How to combine exposure measurements with a job exposure matrix

In order to better estimate occupational exposure to chemicals in the general worker population, I'd like to combine a job exposure matrix (JEM) with chemical exposure measurements. A generic JEM is ...
0
votes
0answers
10 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
12 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
1answer
37 views

Is there a way to reduce high-dimensional feature space to an array of 2d tSNEs ordered along a chosen dimension?

Let's say we have 4096-d vectors (via a CNN fully-connected layer) and often we use tSNE to visualize the space, sometimes in combo with Jonker-Volgenant to assign it to a grid. When applied to image ...
0
votes
0answers
19 views

Sure Independence Screening

Could someone please explain Sure Independence Screening in simple terms. It is proposed in the paper by Fan and Lv: Fan, Jianqing, and Jinchi Lv. “Sure Independence Screening for Ultrahigh ...
3
votes
1answer
38 views

Best strategy to maximize the prediction accuracy when p >> n

I am solving the following classification problem: thousands of features, but only 40 samples (i.e. p >> n) classes are balanced it is not possible to get more data the only thing I am interested in ...
0
votes
0answers
19 views

is random projection a linear or non-linear feature extraction method?

The dimensionality reduction has two different types: feature selection and feature selection. As far as i know, the random projection cannot be a feature selection method. Therefore, is it a linear ...
0
votes
0answers
44 views

High dimensional regression overfitting

Consider the linear regression model \begin{equation} \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{equation} where we assume $\boldsymbol{X}$ is $n$-by-$p$, with $p &...
1
vote
1answer
59 views

State space with lasso

Is it possible to incorporate lasso variable selection in the high dimensional state space model. If yes, is there any code or package available in R
0
votes
1answer
29 views

Euclidean distances in spaces of different dimensions

A modest attempt to illustrate my question: Setup Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, ...
0
votes
1answer
51 views

Application of LASSO , Ridge, PLS in feature selection of spectral data

The meatspec data in faraway package is spectral data with 100 features .(215 *101). Use of LASSO over ridge and PLS gives better performance (RMSE based) But none of the features are removed ( no ...
0
votes
0answers
16 views

Implementation of Compressed Regression (examples?)

I found several articles discussing compressed regression, whether in the bayesian framework or ordinary LS/GLM (1, 2, 3 and others). The idea seems simple, the model becomes $$ Y = \Phi X \beta + \...
0
votes
0answers
37 views

Main differences (problems and mathematics) between traditional statistics and high dimensional statistics

High dimensional statistics seems to be hot nowadays. What are the main differences, in terms of questions and problems it tries to solve, as well as the mathematical tools used, between "traditional"...
1
vote
2answers
54 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
1
vote
0answers
111 views

t-SNE for finding nearest neighbors

I had a question about dimensionality reduction for finding nearest neighbors and was hoping someone could help me out here. Suppose I have good features for images, say penultimate layer features ...
9
votes
1answer
179 views

How do children manage to pull their parents together in a PCA projection of a GWAS data set?

Take 20 random points in a 10,000-dimensional space with each coordinate iid from $\mathcal N(0,1)$. Split them into 10 pairs ("couples") and add the average of each pair ("a child") to the dataset. ...
1
vote
0answers
20 views

margin point of view of logistic regression

I am currently studying Lasso for logistic regression and am using Buhlmann et.al book (Statistics for High Dimensional Data) to understand better. Section In this book on page 3.3.1 they define ...
3
votes
0answers
162 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
1
vote
1answer
27 views

Classification problem for continous target

I have a big dataset with around 100k samples and 2k real-value features. The target variable is in [-1,1], but its distribution is highly concentrated around zero (...
1
vote
1answer
149 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
0
votes
0answers
12 views

non-normal censored outcome, high dimensional predictors

I have an outcome that is non-normal (multimodal) and has an upper limit of detection giving censored values at an upper bound. I have several hundred observations and wish to predict these non-normal ...
2
votes
0answers
28 views

PCA: mean of marginal distribution of high-dimensional vector

Consider the following probabilistic model: $$p(x) = \mathcal{N}(0, I_d), \ x \in \mathbb{R}^d$$ $$p(y|x) = \mathcal{N}(Wx + \mu, \sigma^2I_D), \ y, \mu \in \mathbb{R}^D, W \in \mathbb{R}^{D\times d}...
1
vote
0answers
90 views

algorithms for high dimensional data in a small sample size

I am trying to analysis high-dimensional data in a small sample size and I meet a problem: First,I divided all samples into two sets: training set and test set. Second, in the training set The Y is ...
0
votes
1answer
37 views

Clustering by same random projection

I have $N$, $1024$-dimensional vectors. I want to cluster them by some similarity. Given the high dimensionality, standard metrics won't work. I tried a few Approximate Nearest Neighbor ...
1
vote
1answer
37 views

How to approach preprocessing large number features for machine learning?

I used to apply supervised machine learning for maximum few dozen "normal", natural features like human interpretable ones in Boston House Prices table. I usually try to understand each of them, think ...
0
votes
0answers
32 views

Comparing distances evaluated on different vector spaces

We have a dataset of I items who have been measured over two different sets of features A, with cardinality N, and B with cardinality M, and N > M. We would like to know in which feature space the ...
1
vote
0answers
104 views

MLE for high dimensional $\theta$

I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...