Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
1 vote
0 answers
21 views

How does UMAP deals with the curse of dimensionality?

This question is related to this one which was not answered. The curse of dimensionality states that in high dimension every distance between pairs of points tends to be the same. See this answer for ...
user avatar
  • 111
0 votes
0 answers
29 views

Unbiased estimator of regression coefficient in high dimension

Is there any unbiased estimator for the regression coefficient $\beta \in \mathbb{R}^p$, p >> 1, where $$ y_k = x_k^T\beta + \epsilon \in \mathbb{R}? $$ Note that $x_k \in \mathbb{R}^p$ and $\...
user avatar
0 votes
0 answers
28 views

Clustering high-dimensional, categorical data

I'm trying to cluster ingredients in recipes to see which recipes cluster together. This is a kaggle dataset here: https://www.kaggle.com/datasets/shuyangli94/food-com-recipes-and-user-interactions ...
user avatar
  • 103
2 votes
0 answers
25 views

High dimensional behavior of Dirichlet Process-based clustering?

I have a problem stemming from Dirichlet Process Gaussian Mixture Models (DP-GMMs) in high dimension. I'll write this question so that no knowledge of DP-GMMs is needed. Let $D$ be the dimensionality ...
user avatar
1 vote
0 answers
16 views

Best way to test whether multidimensional data varies mostly in one direction?

I have some multidimensional data. I would like to measure how "one directional" the data is. For example, the rows in this sample ...
user avatar
  • 11
0 votes
0 answers
22 views

Feature selection from large time series data

I have a dataset containing thousands of economic variables (FRED data) and I am looking to algorithmic-ally extract a set of leading indicators to be used in a forecasting framework. Granger ...
user avatar
0 votes
0 answers
20 views

Targeted Learning in Data Science: background material

I am interested in the use of modern causal inference methods to research the association between a (non-genetic) exposure, and endogenous molecules and/or health outcomes (high-dimensional data). I ...
user avatar
0 votes
0 answers
14 views

Formal way to test what kind of differencing is necessary?

I'm working on a project that concerns time series data for South-Africa. My series has 34 explanatory variables and only (!) 30 yearly observations. The analysis is meant to be high-dimensional, ...
user avatar
  • 1
1 vote
0 answers
37 views

How to efficiently do PCA/SVD on dataset with thousands of features (both continuous and OHE) [duplicate]

I am currently dealing with a dataset with about 300,000 records, there are a wide variety of categories in several columns and naturally when one-hot-encoding these the number of features increases ...
user avatar
  • 11
1 vote
1 answer
68 views

Is it wrong to run a Random Forest on high-dimensional, sparse, and unbalanced data?

I am learning about random forests, and I have been testing using R. I have doubts about whether I am doing something wrong given that my data are: sparse, high-dimensional, and unbalanced. Trying to ...
user avatar
4 votes
1 answer
196 views

Wainwright's HDS Exercise 5.5 Gaussian and Rademacher Complexities

This is about upper bounding Rademacher complexity by Gaussian complexity but I am only asking about a step in the proof and the actual question is not so important. A similar question was asked ...
user avatar
0 votes
0 answers
40 views

How many variables can logistic regression handle?

Not from a theoretical point of view, but from a practical one: is there a point at which there are "too many" predictors for a logistic regression model to perform well? For larger datasets ...
user avatar
1 vote
1 answer
159 views

When can I say there is no clusters?

I am trying to do cluster analysis for a very small data set (<100) in higher dimensional feature space. I tried K-means and Hierarchical clustering, but I found no 'elbow' and also the silhouette ...
user avatar
1 vote
0 answers
37 views

What am I missing to perform forward stepwise regression when p > n?

I keep running into warnings in RStudio when I use subsets where p > n. ISLR 6.4.3 mentions that forward stepwise can be useful for high dimensional data, which I'm trying to just test out for ...
user avatar
5 votes
1 answer
100 views

When is a model high-dimensional?

Usually a model is considered to be high-dimensional when $n \ll p$, where $n$ is the number of the observations and $p$ the number of the variables/features (e.g. Bühlmann and van de Geer, 2011). ...
user avatar
  • 327
0 votes
0 answers
22 views

Neural network for mesh data

What I'm trying to achieve: I want to be able to input a mesh into a pipeline and output a reconstructed mesh. I'm having trouble figuring what approach to take and everything I've tried so far isn't ...
user avatar
  • 21
0 votes
0 answers
10 views

Combining multiple datasets vs multiple models in high dimensions

This question is related to this one and this one, but I was wondering about this topic in general. Imagine a setting where multiple datasets, representing different measurements, have been gathered. ...
user avatar
7 votes
4 answers
651 views

Understand the illustration of the curse of dimensionality?

I understand how the curse of high dimensionality works when most features are irrelevant in this highly cited article: A Few Useful Things to Know About Machine Learning, but I get stuck in reading ...
user avatar
  • 5,151
2 votes
0 answers
54 views

How is UMAP a valid dimensionality reduction technique when it uses KNN, which suffers from the curse of dimensionality?

I have not a found a satisfactory, or really any answer, to the following problem that I cannot resolve myself. UMAP is touted as an excellent dimensionality reduction technique by constructing a high-...
user avatar
  • 106
0 votes
0 answers
18 views

Using UMAP on the space of principal components, a valid proposition?

I have a genomics dataset with roughly 16,000 features. Currently, I'm in the process of clustering for cell subtype identification, which I'll then build a classifier on. However, I've run into two ...
user avatar
  • 106
2 votes
0 answers
34 views

Choosing best hyperparameters for multiple regression when number of features is higher than number of samples

I am a chemist mostly, and I do not have much experience in statistical learning. However, I am currently starting work on a problem that requires multiple regression. I have a set of molecules, for ...
user avatar
  • 153
0 votes
0 answers
43 views

Compatibility condition in LASSO

I am reading Statistics for High-Dimensional Data (Bühlmann and van de Geer). Chapter 6 discusses obtaining the oracle inequality in LASSO under the compatibility condition, a technical assumption ...
user avatar
15 votes
1 answer
355 views

What do high dimensional cauchy distributions look like?

A well-known rule of thumb is that for high dimensions $d$, the Gaussian distribution $N(0,I_d)$ is approximated by the uniform distribution on a sphere $U_{\sqrt{d}S^{d-1}}$. This has been mentioned ...
user avatar
  • 1,124
1 vote
0 answers
53 views

Concentration Inequalities for Low-Dimensional Normal Vector

I've got a low-dimensional standard normal vector $g\in\mathbb{R}^n,~~ g\sim\mathcal{N}(0,I_n)$, where $n\le 3$. I want to bound the $L_1$ and $L_2$ norms of $g$ using concentration inequalities (...
user avatar
  • 2,351
0 votes
0 answers
21 views

Hypothesis Testing on Derived Distributions

Suppose we have access to samples from two probability distributions $P$ and $Q$ which may be arbitrary and high dimensional but are over the same domain $\mathbb{X}$ (for example $P$ and $Q$ may be ...
user avatar
0 votes
0 answers
27 views

Does a gaussian kernel suffer from the curse of dimensionality?

Some embedding methods map a data vector in original space to a new space with significantly high dimension and then calculate dot product between these mapped high dimensional vectors. Don't they ...
user avatar
2 votes
0 answers
165 views

Bias-variance trade-off between LDA and QDA w.r.t. dimensionality

Consider the bias-variance trade-off between linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA). Switching from QDA to LDA will generally yield a reduction in variance. The ...
user avatar
0 votes
0 answers
23 views

What predictive model would work best for estimating a single label using time series of varying lengths as features?

I am working on a time series problem for which I have not yet found an obvious solution. The goal is to forecast a metric of patient health using time series of various clinical features. For each ...
user avatar
  • 1
2 votes
1 answer
65 views

Is Individual Coefficient Significance with Ridge or Lasso possible, when Amount of Variables exceeds Observations

First, to introduce you to my situation, I have a dataset containing n = 16 observations and p = 17 variables. My variable set contains 16 independent variables (14 variables I'm interested in and two ...
user avatar
1 vote
0 answers
81 views

Diagonal of the projection of a design matrix with fixed effects

I am looking to compute the diagonal entries of a projection matrix $$ P(X) = X (X' X)^{-1} X' $$ where $X$ is a design matrix that contains high dimensional fixed effects, that is, $X = [A ~~ D]$ ...
user avatar
  • 111
1 vote
0 answers
69 views

How to get the bias-adjusted variance estimates for high-dimensional fixed effects in a linear model?

I want to do a linear regression: $y_{i}=\psi_{j(i)}+X_{i}^{\prime} \xi+\varepsilon_{i}$, where $\psi_{j(i)}$ is a high dimensional fixed effect on group $j$, $X_i$ are the covariables of each ...
user avatar
1 vote
1 answer
101 views

Probability Distributions : "Mode" vs. "Expectation"

I have often heard the argument that in higher dimensions: the "mode" (most common value) of a probability distribution function does not correspond to the "expectation" (mean) of ...
user avatar
  • 5,718
2 votes
2 answers
43 views

How to find the number of clusters when more than one datasets are aggregated as one?

Suppose 3 datasets has 3 ,7, 4 clusters in their respective dataset. When I aggregated them as one dataset what's the safest number of cluster to choose as perimeter for kmeans or any supervised ...
user avatar
11 votes
4 answers
1k views

Dimension reduction using space filling curve to avoid "Curse of dimensionality"?

In machine learning, we want to train a model. While training, if the dimension of data is high, we have a problem (Curse of Dimensionality), so we want to reduce the dimension of our data. Since we ...
user avatar
1 vote
0 answers
27 views

High dimensional multivariate normal calculations in R

Good day everyone At the moment I am attempting to write code in R to calculate the following. $$ \tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^...
user avatar
0 votes
0 answers
21 views

ML algorithm for high dimensional time series forecasting

I'm trying to make a forecasting model for goods prices in an economy (trying to forecast inflation). Dataset: has 300 goods prices % monthly variations for last 6 years. And also added $n$ ...
user avatar
2 votes
1 answer
92 views

When doing hierarchical clustering, do we need to exclude variables with high correlation?

I have one question regarding the hierarchical clustering. I personally have used this hierarchical clustering methods a few times, but did not apply it to the protein level data before. What I am ...
user avatar
0 votes
0 answers
20 views

Issues in having high-dimensional and sparse data

I was wondering about the issues one would encounter in a Machine Learning algorithm having data represented by high-dimensional vectors that are also sparse. In particular, I know that having many ...
user avatar
1 vote
0 answers
49 views

Probabilty estimation for Bernoulli with number of trials as random variable

Problem description Suppose we have fixed number of people that are the test population, let's say $t=200$ persons. For each one of them $\mathbf{r}_j$ we know about $m=300$ features that describes ...
user avatar
  • 11
0 votes
1 answer
84 views

Scaling a gene expression data generates NA values

I would like to analyze the prostate gene expression data which has a link named 12859_2005_967_MOESM4_ESM.tgz in the site here. In a paper I read, the author scaled the predictors in the training set ...
user avatar
  • 263
1 vote
0 answers
38 views

Comparing high dimensional samples

I am working on a project with Convolutional Neural Networks and their internal representation of the classes on the "feature space" (aka. the flatten layer at the end of all the convolution ...
user avatar
  • 71
5 votes
1 answer
364 views

If the curse of dimensionality exists, how does embedding search work?

The curse of dimensionality tells us if the dimension is high, the distance metric will stop working, i.e., everyone will be close to everyone. However, many machine learning retrieval systems rely on ...
user avatar
  • 33.3k
0 votes
0 answers
119 views

Regression in high dimensions ($p>n$) with nonnegative outcome

I need to perform regression in a setting where my outcome is positive ($Y > 0$) and there are more variables than rows ($p > n$). The goal of the analysis is to obtain predictions, which, of ...
user avatar
  • 625
0 votes
0 answers
122 views

Tweedie distributions for large dataset, power parameter for different density peaks and random effects

I have a large dataset (343750 variables and 151 observations) and I want to model each variable as a response one in order to know if it can be explained by the group of patients and the gene ...
user avatar
1 vote
0 answers
34 views

Sampling marginal distribution from joint density

Suppose we know that random vectors $x, y$ have joint density $p(x, y) \propto \exp(-U(x_1, \ldots, x_m, y_1, \ldots, y_n))$, and we want to draw a random sample from the marginal $p(x)$ (i.e. we want ...
user avatar
  • 111
1 vote
0 answers
10 views

Coefficients estimation via covarinace matrices in linear regression

In linear regression I know the OLS-estimator $\beta = (X'X)^{-1}X'y $, but if the number of possible regressors $p$ is high, e.g. higher than the number of observations $N$, this is not a feasible ...
user avatar
0 votes
0 answers
39 views

How to do normality test for high dimension data?

I have samples from a $d$ dimensional distribution $p$. The distribution of $p$ is unknown. I want to use the samples to judge whether or not the $p$ is close to a standard unit Gaussian distribution. ...
user avatar
0 votes
1 answer
258 views

Random forest for high dimensional data with repeated measures (the LongituRF package in R)

I have some high dimensional repeated measures data, and i am interested in fitting random forest model to investigate the suitability and predictive utility of such models. Specifically i am trying ...
user avatar
1 vote
2 answers
204 views

Ways to compare feature selection methods

Context: A hyperspectral image is taken (here Indiana Pines) which needs to be reduced to a lower dimension from 200 bands for this GSA is to be used. What will be possible metrics to grade various ...
user avatar
3 votes
1 answer
93 views

Asymptotic order of the $L_\infty$ norm of asymptotically normally distributed random variables [closed]

Let $\mathbf{X}_n \in R^p$ be a random variable and $\mathbf{s} \in \mathcal{S} = \{x \in R^p \ s.t. \ ||x||_2 = 1\}$. Then, suppose that $p$ and $n$ are allowed to diverge and that we have $$\sqrt{n} ...
user avatar

1
2 3 4 5
7