# Questions tagged [sparse]

A sparse matrix is a matrix where many of the elements are zeros. The tag can also be used for sparsity in other contexts, such as regression models with sparsity, or the "bet on sparsity"-principle.

236 questions
Filter by
Sorted by
Tagged with
19 views

### Correaltion between sparse variables

I have an Events x People matrix M, where a cell (e,p) gives the score of person j at event e. Let E be the total number of events. Each person has attended a lot of events, say 0.3*E at an average. ...
31 views

### How is explained variance in sparse PCA calculated?

Sparse PCA is a technique proposed by Zou et all in this paper. In usual PCA the obtained loadings are orthonormal, and the resulting scores are uncorrelated. However, in sparse PCA you give up these ...
17 views

### Using Pearson correlation coefficient in sparse data

I have been using the cor function in R to compare correlation between my variables. The data did pretty poorly with 2/3 of them having a correlation close to 1 with some other variable. However the ...
160 views

### What does Sparse PCA implementation in Python do?

I am interested on using sparse PCA in python and I found the sklearn implementation. However, I think this python implementation solves a different problem than the original sparse pca algorithm ...
19 views

### How does Data Augmentation work for supervised learning models?

I've ran into a few Kaggle competitions where the winning solution used data augmentation, and a new ML platform, which claimed to help with Data Augmentation. Use cases were imbalanced classes and ...
30 views

60 views

### Efficient way to do Autoencoder on large sparse matrix

I have a large csr_matrix of shape (60,000, 180,000) and about 99.7% sparsity. I was trying to train an autoencoder for this matrix via mini-batch optimization. I tried batch size of 6000 with ...
20 views

### Relationship Between PCA Principal Components & Dictionary Learning Atoms

Suppose I am given an image, where I generate n random 16x16 patches that are each flattened as 256 x 1 vectors, i.e. the number of variables p is 256. Upon performing PCA, I find $min(n, p)$ ...
32 views

### estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
119 views

### Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
393 views

### Autoencoder for sparse data

Suppose I have a big (1,000x20,000) sparse (95% of elements are zeros) matrix with counts. I want to use autoencoder to encode-decode this matrix. How should I do it? Are there any tricks or ...
150 views

### estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
210 views

### LSTM time series forecasting on sparse dataset

I am working on the LSTM time series forecasting of solar energy production. The available data is one year on a half hourly basis. More than 60% of the data values are zero as the PV stations cannot ...
39 views

### Longitudinal study - generalised linear mixed model - dealing with very wide confidence intervals due to sparsity in the outcome

I am conducting a treatment evaluation using administrative data. It is a population-based study of all people diagnosed with a specific disorder in two calendar years (N = 2300). I have run a GLMM ...
21 views

### How much data is considered “sparse” for fitting a mixed (Beta Geometric) distribution with 4 shape parameters?

I'm using CamDavidsonPhillips Customer Lifetime Value library to calculate CLV, and it uses a distribution based on Peter Fader's work on the subject that fits a Gamma distribution to model customer ...
108 views

### Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
16 views

### How to develop features for deep learning from cart items data?

I wonder how to approach building set of features to feed deep learning model (eg convnet) from cart items data: 5pcs of product1 1pcs of product5 2pcs of product8 Assuming 30-50 products per ...