Questions tagged [sparse]
A sparse matrix is a matrix where many of the elements are zeros. The tag can also be used for sparsity in other contexts, such as regression models with sparsity, or the "bet on sparsity"-principle.
292
questions
2
votes
1
answer
130
views
MAE to find tuning parameter for lasso logistic regression
I have a classification problem. The actual outcomes are binary (0 or 1), but I want to predict probabilities, rather than predicting simply 0 or 1. I also want something with feature selection, since ...
- 536
1
vote
1
answer
27
views
What are the modes of a dictionary / transform basis?
So, I'm reading Steven Brunton's book, "Data Driven Science & Engineering", and I'm trying to understand what he means by mode in this following excerpt:
Most natural signals, such as ...
- 13
0
votes
0
answers
15
views
How to measure distances between sparse vectors when the dimensions are not independent?
I am searching for a similarity metric between vectors that are sparse-ish, but most distance metrics treat each dimension as independent and this can create problems when comparing distances between ...
- 1
0
votes
0
answers
13
views
How adding sparsity term in loss function of sparse autoencoder in-actives the hidden node?
I am working on a Sparse Autoencoder but Andrew NG's notes are hard to understand. My question is about the following equation: Loss Function. image
In sparse autoencoder, the goal is to inactive some ...
2
votes
1
answer
64
views
Parameter choice rules for L1 regularization?
I am solving an L1 regularized least squares (LASSO) of the form:
$$ \arg \min_{\boldsymbol{x}} \frac{1}{2} {\left\| A \boldsymbol{x} - \boldsymbol{y} \right\|}_{2}^{2} + \lambda {\left\| \boldsymbol{...
- 21
0
votes
0
answers
9
views
Questions about Dictionary Learning
I have been studying sparse dictionary learning through this Wiki page, and I have a few questions:
In the screenshot below, it says:
$\mathcal{C}$ is required to constrain $\mathbf {D}$ so that its ...
- 303
0
votes
0
answers
133
views
Multilabel Classification: Accuracy is very low. Metric or Model, which is inadequate?
In my multilabel classifaction problem, which I approach similarly to what can be see in this post: How does Keras handle multilabel classification?, the resulting accuracy only increases from 2% to 5%...
- 101
0
votes
0
answers
18
views
Mask in transformers and its relation with sparse attention
Can anyone please explain in a clear way what is the usage of mask in attention for sparse attention? I just can not get how masking tokens (I do not mean here pad tokens) can make attention more fast ...
- 121
1
vote
0
answers
15
views
Sparse Matrix Resulting in Rank Deficient Correlation Matrix
Suppose I have a matrix with 34 binary predictors and a large number of observations such that there are always 5 ones per observation (i.e., 5 of the predictors are randomly set to 1 in each ...
- 11
2
votes
0
answers
26
views
Is the Sparsity Pattern (Number of Non Zero Elements) of the Lasso Solution Monotonic as a Function of $ \lambda$? [duplicate]
For solving:
$$\hat\beta^\lambda = \arg\min_{\beta \in \mathbb{R}^p} \|y - X \beta\|_2^2 + \lambda \|\beta\|_1,$$
We know that the values of $ \beta $ might increase for some $ {\lambda}_{2} > {\...
- 129
2
votes
0
answers
22
views
Recommendation for books about statistical data analysis with sparse data ( a lot of zeros)
Evident from the title, i am looking for books about the general statistical data analysis techniques such as hypothesis testing, etc but for large and sparse dataset.
0
votes
0
answers
30
views
How to choose tuning parameters for SparsePCA
Good morning to everyone.
I'm studying the paper "Sparse Principal Component Analysis Hui ZOU, Trevor HASTIE, and Robert TIBSHIRANI" (Link: https://hastie.su.domains/Papers/spc_jcgs.pdf)
At ...
0
votes
0
answers
12
views
Obtain principal components from the loading vectors
Good morning, everyone.
I am trying to use the "SparsePCA()" function of Matlab whose documentation link is below available.
https://www.ml.uni-saarland.de/code/sparsePCA/sparsePCA.html
This ...
4
votes
2
answers
116
views
Creating a random sparse precision matrix?
In my current project, I want to create a random sparse precision matrix $\boldsymbol{P}=\boldsymbol{\Sigma}^{-1}$ (the inverse of a covariance matrix $\boldsymbol{\Sigma}$). My current procedure ...
- 503
5
votes
2
answers
136
views
How do Sparsity Priors help for Identifiability?
Let's say we have a Factor Analysis model with a latent variable $\mathbf{z}_t \in \mathbb{R}^k$:
$$x_t = A z_t + \epsilon_t, \qquad \epsilon_t \sim \mathcal{N}(0, \Sigma)$$
Let $A \in \mathbb{R}^{g\...
- 57
0
votes
0
answers
15
views
Questions about the exact covariance - matrix and Adjusted Total Variance
Good evening everyone.
Regarding the topic related to Sparse PCA. For example, in "Sparse Principal Component Analysis Hui ZOU, Trevor HASTIE, and Robert TIBSHIRANI" the authors mention the ...
1
vote
1
answer
79
views
Features differ between classes
Good evening everyone.
Regarding the topic related to Sparse Clustering (for example K-Means).
For example, in "Witten DM, Tibshirani R. A framework for feature selection in clustering" the ...
0
votes
0
answers
59
views
Does dataset sparsity affect rate of convergence of model solution?
I have a model with not too many ordinal data. The model performs at a 90% accuracy. I am thinking of adding 13 ordinal variables and transforming them using one hot encoder. The transformation will ...
- 1
1
vote
0
answers
42
views
How to implement simple VAE with sparse tensor in Tensorflow
thank you for reading.
I have been attempting to train a simple VAE on very sparse 2D and 3D data. So far I have been training using dense tensors which - I think - is resulting in horrible training ...
- 11
1
vote
0
answers
47
views
Multivariate time-series with asyncronous data
I have quote observations for 9 FX rates, which I would like to analyze via multivariate dynamic linear model (e.g., Chapter 16 in West and Harrison Bayesian Forecasting and Dynamic Models (1997).
In ...
- 392
1
vote
2
answers
75
views
Does 'sparse' mean different in sparse feature & sparse representation?
In Google developers site, the meaning of the word sparse seems to contradict between the following two definitions:
sparse feature:
Feature vector whose values are predominately zero or empty...
...
4
votes
1
answer
328
views
Car rental time series forecasting
I have the time series of car rentals/demand in a location. The time axis is every hour for 3 months. The y-axis is number of car rentals/demand.
I want to do prediction for future hours given this ...
- 223
0
votes
0
answers
8
views
Item clustering to help with extremely sparse collaborative filtering
I have a large collaborative filtering dataset, where items are images. Some have text, other don't, some are quite visually similar. I do know that some items are similar, e.g. I have 10 distinct ...
- 229
2
votes
1
answer
48
views
How to use sparse PCA loadings in a regression?
I'm using Dr Frank Harrell's code in RMS 2nd edition. He goes into sparse PCA.
Does anyone know how to code a regression model after getting the sparse component grid?
...
- 21
3
votes
0
answers
32
views
Fitting Sparsed Constrained regression with non-negative coefficients adding to 1
I see a similar problem in How do I fit a constrained regression in R so that coefficients total = 1?
Specifically, my model is
$Y_i= \pi_1 X_1+\pi_2 X_2 +...+ \pi_K X_K +\epsilon_i$ with $\pi_k \ge 0$...
1
vote
0
answers
89
views
T-testing on sparse data
Problem: I have two stochastic processes, $S_1$ and $S_2$, that frequently are zero, but occasionally have positive values with unknown probabilities $q_1$ and $q_2$. e.g.
$$
S_1 = \{0,0,0,0,0,21,0,0,...
- 11
1
vote
1
answer
86
views
Why lasso cannot be arbitrarily applied?
Consider any log likelihood function $f(\theta|x)$ where $x$ is data. I can consider $f(\theta|x)+\lambda||\theta||_1$ where $||\theta||_1$ is the standard $L_1$ norm.
It seems that I cannot apply ...
- 1,315
1
vote
2
answers
589
views
How to solve the problem of having sparse data that would become too small when aggregated?
I have a dataset that provides the count of cyber incidents since 2011 for different countries and different attack types, and I want to use this data in a machine learning model to predict future ...
2
votes
1
answer
290
views
Is it wrong to run a Random Forest on high-dimensional, sparse, and unbalanced data?
I am learning about random forests, and I have been testing using R. I have doubts about whether I am doing something wrong given that my data are: sparse, high-dimensional, and unbalanced.
Trying to ...
- 131
1
vote
0
answers
37
views
Working with sparse data in numpy and sklearn [closed]
I have a time series dataset geenrated from some electrophysiological data. I have a frequncy dataset and the matrix is quite sparse but huge, like it contains 0.005 s time bins for 2000+ neurons ...
- 358
0
votes
0
answers
48
views
bootstrapped l2,1 least square does not produce sparse solution
I am trying to model an autoregressive model with $\ell$-2,1 regularization, where $\|X\|_{2,1}=\sum_i|\sum_jX_{ij}|$:
$$y_{t+1} = wx_{t} + \lambda_1\|w\|_{2,1}, y\in \mathbb{R}^{n_1}, x\in \mathbb{R}^...
- 303
0
votes
1
answer
127
views
Variable selection with sparse data
I have a dataset with 141 observations and 8 corresponding variables and I mean to apply a GLM to this dataset. However, a lot of observations lack either one or multiple variable values. So if I want ...
- 11
1
vote
1
answer
26
views
How to compare row entries in a sparse table with lots of missing values?
I have a dataset with ~1000 laptops and performance results across ~100 different benchmarks. Using the benchmark results, I want to give each laptop a single composite performance score, and rank the ...
- 113
4
votes
1
answer
321
views
Autoencoder doesn't learn 'sparse' input images
I am trying to train an autoencoder with PyTorch on 2D images containing 2D Gaussian densities such as this:
The images are of size 100x100 (I feed them into the autoencoder as 1x10000 tensors). The ...
- 41
1
vote
1
answer
202
views
What is the difference between network sparsification and model pruning
What is the difference between network sparsification and model pruning? I watched USENIX ATC '21 - Octo: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny (at 01:29sec) ...
- 141
2
votes
0
answers
58
views
What are the references for different active set selection methods for sparse Gaussian processes?
I am comparing the different sparse Gaussian process approaches within the fitrgp function in Matlab, but I am struggling to find references for the different choices within the function for the ...
- 21
0
votes
0
answers
79
views
Oracle inequality for LASSO
In chapter 6 of Statistics for High Dimensional Data (Peter BühlmannSara van de Geer), they focus on the normal linear regression model with fixed regressors and begin by stating OLS achieves a risk
$\...
- 503
2
votes
1
answer
34
views
Bayes prior in MAP estimation corresponding to $\ell^0$ penalization
I gather that in the context of penalized least squares, we can interpret a penalty term as corresponding to a prior $\pi(\beta)\propto \exp\{-\text{pen}\}.$
Is this also true for $\ell^0$ ...
- 503
3
votes
1
answer
283
views
Bayesian priors associated with regularization penalties
I gather that adding a penalty term to (linear) least squares minimization typically corresponds with choosing some prior for Bayes estimation in the normal linear regression model. A couple questions ...
- 503
0
votes
0
answers
27
views
LASSO with $L_p$ norms for $1 < p < 2$?
For the sparse linear regression problem, minimizing the LASSO objective $\| X \beta - Y \|_2^2 + \lambda \| \beta \|_1$ is known to recover the optimal data generating parameter $\beta^*$ with the ...
- 101
2
votes
1
answer
154
views
If there is a sparse solution then is the smallest l1 norm solution at least as sparse?
Consider the linear equation $Ax=b$ where $A$ is a matrix and $b$ and $x$ are vectors. Suppose there exists a vector $x_S$ that solves this equation ($Ax_S=b$) and $x_S$ has $k$ entries of value $0$ (...
0
votes
0
answers
41
views
Multiple regression with sparse X
I would like to run the multiple regression:
$$y = X \beta + \epsilon \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,(1)$$
with:
$n \sim 2 \times 10^6$
$p \sim 10^4$
$X$ sparse
70% of rows have 1 non-...
2
votes
1
answer
91
views
In a sparse reward problem, is it possible to remove reward shaping once the RL agent trains long enough to consistently reach the final reward?
I'm new to machine learning, and primarily looking at it from the perspective of it's applications to control theory.
In the application found in this paper, a RL agent attempts to land a spacecraft ...
- 43
0
votes
0
answers
127
views
LARS package in R with specific lambda value
I am trying to use the LARS package in R to obtain a Lasso estimate of the sparse coefficient vector, say $\hat{\beta}_{\text{sparse}}$, as opposed to a coefficient path. In other words, I am not ...
- 1,102
1
vote
0
answers
92
views
outlier detection for sparse data in categorical variable
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
- 11
0
votes
0
answers
70
views
Calculate Data Sparsity - There are no 'zeros'
I am being asked to provide the percentage of data sparsity in my dataset. The challenge I am running into is that there isn't traditional sparsity in my data, no 'zeros', 'NAs', '-' etc.. I am ...
- 13
3
votes
0
answers
410
views
Clustering text embeddings: TF-IDF + BERT Sentence Embeddings
I am trying to cluster a few thousand forum posts that are similar in content to Stackoverfow.
So far, I have tried two main approaches to represent the posts:
TF-IDF
Sentence embedding based on BERT....
- 41
3
votes
1
answer
717
views
Scaling a sparse matrix
I want to apply sparse PCA to a sparse matrix. I was wondering if scaling to mean 0 and unit variance would be appropriate given that my input is sparse?
- 31
9
votes
1
answer
332
views
Probability distribution associated with nuclear norm?
The $\ell_1$ norm of model parameters is often added to loss functions because it induces sparsity in the solution of the overall cost function:
$$ c(\theta) = -\log L(x|\theta) + \lambda ||\theta||_1$...
- 2,916
2
votes
0
answers
93
views
Are Spiking Neural Networks The Next Big Thing? [closed]
Intel recently announced their Loihi chip as part of their "Neuromorphic Computing" research, which is optimized for spiking neural networks (SNNs). I found an example of a problem in which ...
- 423