Questions tagged [sparse]

A sparse matrix is a matrix where many of the elements are zeros. The tag can also be used for sparsity in other contexts, such as regression models with sparsity, or the "bet on sparsity"-principle.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Sparse time series of non negative monthly defect percentage [on hold]

I want to forecast monthly (5th business day of month, no timestamp) defect percentages for multiple states of USA. Data is sparse (with many zeros, not geospatial) and could be any percentage between ...
1
vote
2answers
35 views

What are the benefits of sparse representations and sparse parameters?

What are their benefits? I know sparse parameters are a different story than sparse representations, but I want to know how each of these can benefit us and which one is more important than the other ...
0
votes
0answers
13 views

How to get adjusted explained variance of PC using Sparse PCA in R

I am using "elasticnet" package to do the sparse PCA. However I could only get the percent of explained variance of each PC. I am wondering can I get the adjusted explained variance of each PC using "...
0
votes
0answers
37 views

Extracting the right summary statistics from zero-inflated data sets (i.e. a sparse matrix where everything non-zero is a statistical outlier)

I'm a consumer tech startup founder with rudimentary background in statistics. I need help in processing a large, sparse matrix. I'm logging all actions users are undertaking in my app. I then ...
0
votes
0answers
26 views

Comparing clusterings of different spaces

I have developed a graph embedding method that learns feature representations for each node in the graph based on the graph topology. I want to demonstrate that the learned feature vectors yield ...
2
votes
0answers
26 views

How to interpret the learning curve of a LinearRegression for sparse data?

I have a dataset of shape ca.(4800, 350). Both the dataset X as well as the response y is very sparse (ca. 3500 samples with y=0). I wanted to take a look at the learning curve to estimate the bias-...
3
votes
1answer
96 views

Sparse linear poorly constrained least-squares problem

I have a somewhat simple linear problem. I have data $D$ (a vector with a few million elements), the parameter vector $X$ (a couple of thousands elements) and the design matrix $A$ which is extremely ...
3
votes
1answer
56 views

Combining many sparse binary variables

Based on kjetil b halvorsen suggestion, I rephrased my problem: My problem is analogous to the following: i am supposed to predict if a high school student will go to university (Yes/No). I have ...
0
votes
1answer
24 views

Nonlinear approximation of a sparse linear transformation

Suppose that we have $y = Ax$ where $x$ is a vector of size $m \times 1$, $A$ is a sparse matrix with size $n \times m$. Suppose that $n \times m$ is very large and we wish to find a non-linear ...
0
votes
0answers
30 views

How to build a machine learning model from sparse features with class imbalance?

I have around 10 numerical features and 1 class/target (e.g., visitors count of a website). All of them are sparse. Sparsity is around %70-80. The median of the class/target is zero. Is there a good ...
0
votes
0answers
8 views

Appropriate machine learning technique for spectral data and low-frequency feedback

I have a performance measure and a data source that basically supplies a complex and varying waveform. I would like to apply some unstructured learning technique to try and find a pattern in the ...
0
votes
1answer
74 views

Clustering Binary and Continuous Features

If you need to cluster a dataset with the following characteristics: It has a mix of binary and continuous features. It is very sparse. For most features, you only have values for 15% of the ...
0
votes
0answers
25 views

Why neural network does not learn sparse relationship [duplicate]

The universal approximation theorem says that a neural network can be used to approximate any continuous function under some regular conditions with arbitrarily small approximation error, provided we ...
2
votes
0answers
20 views

Probability of sparse spectrum

Consider a vector $v$ such that $v \sim \mathrm{Unif}(\mathbb{S}^{d-1})$, the uniform distribution on the unit sphere in $d$ dimensions. Question: is there an upper bound on the probability that $v$ ...
1
vote
0answers
32 views

Is it useful to use sparse regression (e.g. Lasso) when the number of observations is significantly larger than the number of covariates?

I'm learning about penalized/sparse regression and I noticed that the examples used for penalized/sparse regression, e.g. Lasso, are usually cases where the number of observations is significantly ...
2
votes
0answers
59 views

How to conduct a principal component analysis on data set with large number of zeros

I have data for percentage cover of plant species in 500 sites. There are columns for 30 different species in the data set and I would like to drastically reduce this down to a manageable number of ...
0
votes
0answers
23 views

Notation problem with sparse regularized correlation matrix

I am trying to apply a specific method to obtain a sparse correlation matrix $R$ from a regularized correlation matrix $\Sigma^{\delta}$, which was computed from $N$ observations of a multivariate ...
0
votes
0answers
7 views

Regress in two variables against basis functions when one of them might be “sparse”

I'm tasked with the problem of fitting a series of functional forms $\{f_t(S,K)\}$ along the time axis. At each time $t=0,1,\cdots,T$, there are $N_t$ samples $$(F_{t,i}, S_{t,i}, K_{t,i}),\quad i=1,2,...
0
votes
0answers
43 views

Efficient way to do Autoencoder on large sparse matrix

I have a large csr_matrix of shape (60,000, 180,000) and about 99.7% sparsity. I was trying to train an autoencoder for this matrix via mini-batch optimization. I tried batch size of 6000 with ...
0
votes
0answers
15 views

Relationship Between PCA Principal Components & Dictionary Learning Atoms

Suppose I am given an image, where I generate n random 16x16 patches that are each flattened as 256 x 1 vectors, i.e. the number of variables p is 256. Upon performing PCA, I find $min(n, p)$ ...
0
votes
0answers
28 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
2
votes
0answers
113 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
0
votes
2answers
170 views

Autoencoder for sparse data

Suppose I have a big (1,000x20,000) sparse (95% of elements are zeros) matrix with counts. I want to use autoencoder to encode-decode this matrix. How should I do it? Are there any tricks or ...
3
votes
1answer
123 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
1
vote
1answer
140 views

LSTM time series forecasting on sparse dataset

I am working on the LSTM time series forecasting of solar energy production. The available data is one year on a half hourly basis. More than 60% of the data values are zero as the PV stations cannot ...
0
votes
1answer
34 views

Longitudinal study - generalised linear mixed model - dealing with very wide confidence intervals due to sparsity in the outcome

I am conducting a treatment evaluation using administrative data. It is a population-based study of all people diagnosed with a specific disorder in two calendar years (N = 2300). I have run a GLMM ...
0
votes
0answers
17 views

How much data is considered “sparse” for fitting a mixed (Beta Geometric) distribution with 4 shape parameters?

I'm using CamDavidsonPhillips Customer Lifetime Value library to calculate CLV, and it uses a distribution based on Peter Fader's work on the subject that fits a Gamma distribution to model customer ...
6
votes
1answer
95 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
0
votes
0answers
24 views

Sparse solutions: linear systems vs logistic regression

It is known in the field of compressed sensing/sparse approximations that if $$Ax = b$$ has sparse solution (with $s$ nonzeros), then there is a condition which states that it is unique, if $$s \leq 0....
0
votes
0answers
41 views

Sparse coding and feature learning

Recently I tried to understand sparse coding and its application to classification. But there is no way to check whether I understand correctly, so I have a few questions about this algorithm. I ...
0
votes
0answers
16 views

How to develop features for deep learning from cart items data?

I wonder how to approach building set of features to feed deep learning model (eg convnet) from cart items data: 5pcs of product1 1pcs of product5 2pcs of product8 Assuming 30-50 products per ...
1
vote
0answers
32 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
1
vote
1answer
33 views

finding sparse regions in time series data

I have several hundred years of church baptisms that will be searched by people wanting to find the baptisms of their ancestors. I want to call attention to periods in the records in which the number ...
2
votes
1answer
2k views

Do you standardize the data before PCA whitening?

I have a data set ranged in different scales as well as some variables are sparse, for example, ...
1
vote
0answers
74 views

Normalizing sparse matrix by mean, should the mean be calculated excluding zero?

I have very sparse matrix (70% sparsity) which I want to normalize by mean. I tried using mean both include and exclude zero. The histogram between count (y-axis) and value (x-axis) shows The value ...
1
vote
0answers
201 views

GAN and NN for sparse data

I have a set of images which represent some correlated sparse data $x_1,\ldots ,x_n$. there are a number of specific pixels in the images which might hold value or not (with some probability), while ...
2
votes
1answer
214 views

Simulation of low rank and sparse matrix

I am having trouble simulating a matrix which is low rank and sparse (sparse along both rows and columns). One way to simulate a low-rank matrix is by generating a random matrix, then taking SVD and ...
5
votes
1answer
2k views

Can it be over fitting when validation loss and validation accuracy is both increasing?

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, ...
3
votes
1answer
167 views

Amount and sparsity of data for recommender systems

I'm starting to work in a project that will have a recommender system as one of its components. I'm trying to figure out if I have the right type of data for the recommender. The data contains ...
1
vote
0answers
66 views

Confidence region for multinomial distribution with k=7 - including 5 zero values in dataset

I have a multinomial distribution with $k=7$ and an observed dataset $n_i = \{62, 35, 0, 0, 0, 0, 0\}$. While it was quite expected that $n_i = 0$ for $i \in \{3,4,...,7\}$, there was no way to ...
3
votes
0answers
28 views

How to prove oracle properties in Fan and Li (2001) paper

I am studying Fan and Li's 2001 paper "Variable selection via nonconcave penalized Likelihood andits oracle properties" but I am having troubles understanding Theorem 1 proof (page 1359). I follow the ...
2
votes
1answer
204 views

Data sparsity becomes a problem

I read this in a paper Developing an approach toward virtual synthesis parameter screening introduces two primary computational challenges: data sparsity and data scarcity. ...Such canonical ...
0
votes
1answer
16 views

Question about minimax risk sparse sparse condition

I am studying statistical leaning theory. Especially the paper "Minimax rates of Estimation for High Dimensional Linear regression over $l_q$ balls" by Garvesh Raskutti .et.al. In the right end of ...
0
votes
0answers
38 views

Is max. Eigenvalue of k-sparse PCA always $\leq$ max. Eigenvalue of normal PCA on same dataset?

Is max. Eigenvalue of k-sparse PCA always less than or equal to the max. Eigenvalue of normal PCA on same dataset? K refers to the number of non zero eigenvalues when the dataset is of dimension n <...
1
vote
2answers
1k views

How to do dimension reduction with sparse data

I have 200 vectors representing the percentage marks for 200 different students in the different classes they took. The vectors are 22 dimensional (as there were 22 different classes in total) even ...
5
votes
1answer
110 views

Covering the unit sphere with sparse vectors

I'm looking for references for covering the $d$-dimensional unit sphere $$ \mathbb{S}^{d - 1} = \left\{ x \in \mathbb{R}^d : \| x \| = 1 \right\} $$ I'm trying to cover $\mathbb{S}^{d-1}$ with ...
1
vote
0answers
38 views

Correlation vs. logit coefficient

I am analysing a data set with a dichotomous independent and a dichotomous depenent variable. Persons r only is r = 0.047. However, if I use the same variables in a logistic regression the logit-...
2
votes
0answers
232 views

How to transform my sparse count data into normal distribution?

I am running glm on beetle counts data. My predictors are environmental variables and my response variable is the number of beetles. I ran three glms: The response variable $Y_1$ is the total number ...
2
votes
2answers
191 views

Database-friendly random projections with Numpy

In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
3
votes
2answers
759 views

Eligibility Traces vs Experience Replay

I am currently using the OpenAI Baselines implementation of DeepQ (paper found here). I am also utilizing Prioritized Experience Replay (paper found here). My problem involves sparse delayed rewards ...