Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
4
votes
0answers
75 views

PCA: inference on the proportion of explained variance, in a large p setting

I am interested in doing inference on the proportion of total variance explained by the first principal component, for a PCA based on the correlation matrix R. I want to know the (asymptotic) ...
2
votes
1answer
114 views

Why does the condition number of the covariance matrix explode as number of variables increases?

From asset returns of $N$ stocks, the symmetric covariance matrix sized $N\times N$ is constructed, which treats the asset returns as variables. When the number of variables $N$ is fairly low like $N=...
3
votes
0answers
31 views

What is the appropriate metric for determining distance / dissimilarity of sparse, high dimensional data in PCA space?

I'm working with scRNA-seq data (~96% sparse, high dimensional), and am trying to determine distances between the cells in PCA space - NOT for the specific purpose of clustering. The principal ...
2
votes
0answers
21 views

Why does augmenting the covariance matrix with additional data not improve out of sample prediction?

I was thinking about the structure of high dimensional regression models, and there are essentially two statistical steps, estimating covariance matrix of the data, and estimating the individual ...
1
vote
1answer
30 views

Does SVM suffer from curse of high dimensionality? If no, Why?

While I know that some of the classification techniques such as k-nearest neighbour classifier suffer from the curse of high dimensionality, I wonder does the same apply to the support vector machines ...
0
votes
1answer
42 views

Curse of Dimensionality - Why does “each variable added result in an exponential decrease in predictive power?”

I read this in this blog. So I'm assuming by variable they mean by adding an extra dimension. How does adding a variable exponentially decrease predictive power? What is predictive power? Is there a ...
0
votes
0answers
20 views

Factor Analysis with more variables/parameters than observations/samples

I have a dataset containing ~150 observations/rows, and significantly more variables/columns (~1000) and would like to perform factor analysis on the data. I had one question on a theoretical level ...
0
votes
0answers
4 views

How to find the best or optimal parameter value set for agent based or other kind of modeling?

I am doing agent based modeling (ABM) for infectious disease modeling, and the model has 50+ parameters which can any probability value between 0 and 1, or can be any float value from 0 to >0 (...
0
votes
0answers
32 views

Data with multicollinearity and p >> n

This is a csv file, the file is titled "res_final". The first line contains the names of the variables: ...
0
votes
3answers
55 views

Suggestions on binary classifiers for high dimensional categorical data set?

I have a binary classification problem with 210 variables (2 levels 0/1) and I am wondering how should I approach this problem as algorithms which I used (logistic regression, random forests) did very ...
2
votes
2answers
38 views

Laplace approximation in high-dimensions

Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
4
votes
0answers
27 views

Prove or disprove a concentration result of the norm of high dimensional random vector

Suppose that $X = (X_1,X_2,\cdots ,X_n)$ is a vector, where $X_i, i=1,2,\cdots ,n$ are independent and sub-gaussian random variables satisfying $\mathbb{E}[X_i^2] = 1$. Prove or disprove the following ...
1
vote
0answers
26 views

PCA all variables vs PCA subgroups

I have about 600 variables (no response variable) and I would like to use PCA to reduce the number of dimensions (variables). After reduction, I only want to have 100 variables. One approach I can do ...
0
votes
0answers
12 views

Explaining cluster by original variables

I have high-dimensional data on observations of individuals. To simplify and visualize the problem let's assume I have the probability of a positive reaction to 10different marketing campaigns of 8 ...
3
votes
0answers
47 views

Why can't we accurately compute covariance matrix in high dimensions?

I am reading pg 651 of Elements of Statistical Learning,where is says: "The simplest form of regularization assumes that the features are independent within each class, that is, the within-class ...
0
votes
1answer
63 views

Comparing clustering of standard errors between felm and feols functions

I'm using the lfe and fixest packages to run regressions with high-dimensional fixed effects. For these regressions, I would ...
1
vote
2answers
60 views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
3
votes
1answer
65 views

Potential applications of a result (by Beyer et al) on distance concentration and meaningless nearest neighbors in high dimensions

My question is motivated by this question, and self-study of the paper "When is nearest neighbor meaningful?", where the authors show the following Theorem 1: Let $X^{(d)} \in \mathbb{R}^d$ ...
0
votes
0answers
13 views

Clustering & classification of customers

I have three datasets : one about general population, one about customers for a specific brand and then one with people that were part of an advertisement event and whether or that person converted to ...
0
votes
1answer
47 views

Use of the term 'dimension' in word-embeddings or various other tensors in AI

I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a ...
3
votes
1answer
44 views

In KNN, why does the number of training examples needed to learn a decision boundary increase (exponentially) as the number of dimensions increases?

In book I'm reading the following is said on k-nearest neighbour algorithms: "As the number of dimensions goes up, the number of training examples you need to locate the concept's frontiers goes ...
0
votes
0answers
8 views

Representing partitioned N-dimensional input data

I am working on N-dimensional inputs and single dimensional response data sets. I want to get local picture of slopes in my response and hence, I am dividing my input space into local regions by ...
0
votes
0answers
24 views

How to compare two high dimensional data samples

I have an anomlay detection problem in high dimensional data. I am using an isolation forest to detect anomalies in this data in an unsupervised manner. I have come across two samples in my data ...
0
votes
0answers
48 views

How does scikit-learn handle high dimensionality in its Gaussian Mixture Model implementation?

I have a dataset of 50,000 rows that I plan to fit with scikit-learn's GMM model. The dataset has 15 features, therefore I treat each row as a vector in the space $\mathbb{R}^{15}$. My question is, ...
1
vote
0answers
21 views

Mean field variational inference in high dimensions

Is it expected that independence assumption done in mean field variational inference is going to yield on average better approximations for low-dimensional models compared to high-dimensional ones?
1
vote
0answers
16 views

Incomplete beta function increasing in $\alpha$

Let $I_{x}(\alpha,\beta)$ is the regularized incomplete $\beta$ function, aka the cdf for a random variable with distribution $\text{Beta}(\alpha,\beta)$. In a recent paper (https://arxiv.org/abs/...
0
votes
2answers
129 views

Linear Discriminant Analysis have Small Sample Size problem (SSS) is it n<<d

It is said that LDA has a Small Sample Size problem (SSS) This problem arises whenever the number of samples is smaller than the dimensionality of the samples. (Source: Chen, L.F., Liao, H.Y.M., ...
4
votes
2answers
113 views

How is typical set defined for general high-dimensional distributions?

I'm only aware of the definitions in Elements of Information Theory, which deal with iid and stationary ergodic processes. From there we can speak of the typical set of, e.g., a high-dimensional ...
0
votes
0answers
15 views

Gradient clipping in high vs low dimensions

Is there a qualitative difference between (l2-norm based) clipping a gradient in a low vs a high dimensional statistical model? In which regime is clipping operation expected to work better?
1
vote
1answer
19 views

Numerical problems with high dimensional multivariate normal distributions

Is there a way around the numerical problem of calculating the denominator of the Multivariate Gaussian distribution of high dimensional vectors? Given the following formula $$f_{\mathbf X}(x_1,\ldots,...
1
vote
2answers
165 views

How to distinguish the continuous and categorical variable based on the number of unique values?

I have a very large dataset contained 600,000 rows and 400 columns. Most of the variables are anonymous and named as Vi, i=1,2,...,400, and all of them are either int or float. When I am dealing with ...
0
votes
0answers
62 views

Curse of Dimensionality: Identifying Correct Statement

Im trying to identify which one of these statements about the curse of dimensionality is correct: a) It means that the performance of the KNN classifier gets worse when the number of predictor ...
0
votes
0answers
10 views

What multivariate distributions (or more generally computational architectures) have easily computed marginals besides multivariate Gaussian?

There is some property of marginals being in the same family that makes this easy. Is this generalizable? For exponential family? Copulas. Looking for computation tractability here not analytical ...
1
vote
0answers
12 views

Is there any manifold learning literature where the both the ambient dimension and intrinsic dimension of the data are high?

I'm new to manifold learning, and from my understanding it normally denotes a subject which starts with the assumption that the data $\{x_1 \dots x_n\} \subset (M,g) \subset \mathbb{R}^p,$ where $(M,g)...
0
votes
0answers
7 views

Low values of silhouette index

I get very low values of silhouette index (0.3) even when external evaluation values are perfect (ARI=1). What could be the reason? Only thing that I could think of, that is specific for this ...
0
votes
0answers
13 views

Is nearest neighbor in low dimensional setting is still the nearest in high-dimensional setting?

I have developed a semi-supervised outlier detection algorithm using K-nearest neighbor approach, where I obtain statistical distributions of Euclidean distances for K-neighbors. I wanted to test it ...
1
vote
1answer
27 views

Anomaly detection on high dimensional Data using k means/SVM/LOF?

I am working on one Anomaly detection problem (unsupervise problem) Data set have 1) 15 columns and around 8k rows , including normal and abnormal(outlier ) rows, without label , all are numeric ...
0
votes
0answers
111 views

How to estimate high dimension fixed effects (HDFE) in a Tobit model?

I'm searching for a possibility to estimate a Tobit model with a high number of individual fixed effects. To be more precise, my goal is to estimate the individual determinants of the number of hours ...
0
votes
0answers
19 views

Negative Binomial Regression Shrinkage Priors P greater than N case

Can someone suggest some literature review paper or any literature regarding Negative Binomial Regression with Shrinkage priors particularly P>N case? P: no. of variables, N: no. of observations in a ...
1
vote
0answers
35 views

Is the coefficient of determination defined for higher dimensional dependent variables?

The coefficient of determination has a clear definition for scalar dependent variables. See for instance the definition here: https://en.wikipedia.org/wiki/Coefficient_of_determination Can this ...
3
votes
0answers
36 views

In Multidimensional Scaling (MDS), is it safe to assume that the optimal embedding dimension grows with the growth of sample size?

My question is more of a theoretical nature, so it'd be great to have some references to papers, but it'd be also nice to see some experiments. Let $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i....
11
votes
2answers
613 views

Mathematical demonstration of the distance concentration in high dimensions

I know that in high-dimensional space, the distance between almost all pairs of points has almost the same value ("Distance Concentration"). See Aggarwal et al. 2001, On the Surprising Behavior of ...
0
votes
0answers
28 views

Visualization of high dimensional data on grid

I am interested in some analysis of a high dimensional domain (d=32). In a 2 dimensional domain, I construct a grid, evaluate given function over the domain and either scatter plot the points or plot ...
0
votes
2answers
34 views

Does a high-dimensionality of a time series refer to its length, number of variables or both?

I'd like to cluster some time series that describe a flow of a variable (say, temperature) throughout a day. Measurements are made every 5 minutes so each time series has 288 values. Are we talking ...
1
vote
0answers
55 views

Sparse PCA for $p >> n$ solution with Elastic Net

I was reading about the sparse principal component approach by Zou, Hastie and Tibshirani but I do not quite understand how they handle the $p \gg n$ case in their paper. To derive the sparse axis, ...
1
vote
0answers
20 views

Visualise high dimensional data - returns of n-asset portfolio as 2D plot, like heatmap?

It's possible to visualise both asset allocations and returns of 2-assets portfolio as 2D heatmap. Like in image below, the visualisation of $[Gold, Silver]$ portfolio with restriction $Gold + ...
0
votes
0answers
92 views

Mean shift clustering and the curse of dimensionality

I've often come across resources that mention that mean shift based clustering doesn't work well in higher dimensions. The sources are as follows: Page 1 of https://www.ncbi.nlm.nih.gov/pmc/articles/...
0
votes
0answers
46 views

Can we always perform SVD on the data matrix before doing high-dimensional logistic regression?

So I'm using lasso logistic regression to classify my data. My data matrix $X$ has dimension $n\times p$ for $p >> n$. As $p$ is on the order of a billion, I expect to face some computational ...
1
vote
1answer
216 views

using finite difference to estimate high dimensional gradient in gradient descent methods

I'm not very familiar with optimization problem, but I know that if the gradient of function is hard to find, it can use finite difference method to estimate it. Like scipy.minimize, it would use this ...
0
votes
0answers
14 views

Visualization and two-group statistical testing in high-dimension binary data

I am planning a biomarker discovery experiment, and the data is expected to take the form of a high-dimensional dataset where each data point is described by 8 binary variables indicating the presence ...

1
2 3 4 5 6