Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sparse]

A sparse matrix is a matrix where many of the elements are zeros. The tag can also be used for sparsity in other contexts, such as regression models with sparsity, or the "bet on sparsity"-principle.

0
votes
0answers
6 views

Efficient way to do Autoencoder on large sparse matrix

I have a large csr_matrix of shape (60,000, 180,000) and about 99.7% sparsity. I was trying to train an autoencoder for this matrix via mini-batch optimization. I tried batch size of 6000 with ...
0
votes
0answers
8 views

Relationship Between PCA Principal Components & Dictionary Learning Atoms

Suppose I am given an image, where I generate n random 16x16 patches that are each flattened as 256 x 1 vectors, i.e. the number of variables p is 256. Upon performing PCA, I find $min(n, p)$ ...
0
votes
0answers
17 views

estimate sparse localized whitening transformation

This is a follow-up to estimate precision matrix with given spatial sparsity pattern, expanding on the second part of that question and formulating more precisely using material from the answer by ...
2
votes
0answers
103 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
1
vote
1answer
92 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
0
votes
1answer
34 views

LSTM time series forecasting on sparse dataset

I am working on the LSTM time series forecasting of solar energy production. The available data is one year on a half hourly basis. More than 60% of the data values are zero as the PV stations cannot ...
0
votes
1answer
23 views

Longitudinal study - generalised linear mixed model - dealing with very wide confidence intervals due to sparsity in the outcome

I am conducting a treatment evaluation using administrative data. It is a population-based study of all people diagnosed with a specific disorder in two calendar years (N = 2300). I have run a GLMM ...
0
votes
0answers
10 views

How much data is considered “sparse” for fitting a mixed (Beta Geometric) distribution with 4 shape parameters?

I'm using CamDavidsonPhillips Customer Lifetime Value library to calculate CLV, and it uses a distribution based on Peter Fader's work on the subject that fits a Gamma distribution to model customer ...
5
votes
1answer
76 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
1
vote
0answers
77 views

Random Forest Regression with sparse data in Python

I am working on a Random Forest regression model to predict housing prices. I have about 500k rows of data with the following information: 1.House area in square meters. 2.Number of rooms. 3.City. ...
0
votes
0answers
17 views

Sparse solutions: linear systems vs logistic regression

It is known in the field of compressed sensing/sparse approximations that if $$Ax = b$$ has sparse solution (with $s$ nonzeros), then there is a condition which states that it is unique, if $$s \leq 0....
0
votes
0answers
28 views

Sparse coding and feature learning

Recently I tried to understand sparse coding and its application to classification. But there is no way to check whether I understand correctly, so I have a few questions about this algorithm. I ...
0
votes
0answers
16 views

How to develop features for deep learning from cart items data?

I wonder how to approach building set of features to feed deep learning model (eg convnet) from cart items data: 5pcs of product1 1pcs of product5 2pcs of product8 Assuming 30-50 products per ...
1
vote
0answers
32 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
1
vote
1answer
26 views

finding sparse regions in time series data

I have several hundred years of church baptisms that will be searched by people wanting to find the baptisms of their ancestors. I want to call attention to periods in the records in which the number ...
2
votes
1answer
844 views

Do you standardize the data before PCA whitening?

I have a data set ranged in different scales as well as some variables are sparse, for example, ...
0
votes
0answers
32 views

Normalizing sparse matrix by mean, should the mean be calculated excluding zero?

I have very sparse matrix (70% sparsity) which I want to normalize by mean. I tried using mean both include and exclude zero. The histogram between count (y-axis) and value (x-axis) shows The value ...
1
vote
0answers
117 views

GAN and NN for sparse data

I have a set of images which represent some correlated sparse data $x_1,\ldots ,x_n$. there are a number of specific pixels in the images which might hold value or not (with some probability), while ...
2
votes
1answer
141 views

Simulation of low rank and sparse matrix

I am having trouble simulating a matrix which is low rank and sparse (sparse along both rows and columns). One way to simulate a low-rank matrix is by generating a random matrix, then taking SVD and ...
5
votes
1answer
986 views

Can it be over fitting when validation loss and validation accuracy is both increasing?

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, ...
2
votes
1answer
77 views

Amount and sparsity of data for recommender systems

I'm starting to work in a project that will have a recommender system as one of its components. I'm trying to figure out if I have the right type of data for the recommender. The data contains ...
0
votes
0answers
12 views

What kind of sparse matrix representation is this?

I am putting together a wrapper for a quadratic programming library. I am going through the C example here but I don't understand the indexing used for the matrices. The relevant excerpt is below, ...
1
vote
0answers
48 views

Confidence region for multinomial distribution with k=7 - including 5 zero values in dataset

I have a multinomial distribution with $k=7$ and an observed dataset $n_i = \{62, 35, 0, 0, 0, 0, 0\}$. While it was quite expected that $n_i = 0$ for $i \in \{3,4,...,7\}$, there was no way to ...
0
votes
0answers
104 views

Getting explained variance from Scikit's SparsePCA model?

I am trying to apply SparsePCA by Scikit: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.SparsePCA.html#sklearn.decomposition.SparsePCA But I don't know how to extract the ...
3
votes
0answers
19 views

How to prove oracle properties in Fan and Li (2001) paper

I am studying Fan and Li's 2001 paper "Variable selection via nonconcave penalized Likelihood andits oracle properties" but I am having troubles understanding Theorem 1 proof (page 1359). I follow the ...
-1
votes
0answers
30 views

Can Chi-Sq Test be used to explore the independence of categorical variables on response variable?

I am investigating the independence of some categorical variables (apps, campaigns...) on the response variable (click through rate). I have some doubts on using Chi-Sq Test here: Chi-Sq Test can be ...
1
vote
1answer
89 views

Data sparsity becomes a problem

I read this in a paper Developing an approach toward virtual synthesis parameter screening introduces two primary computational challenges: data sparsity and data scarcity. ...Such canonical ...
0
votes
1answer
14 views

Question about minimax risk sparse sparse condition

I am studying statistical leaning theory. Especially the paper "Minimax rates of Estimation for High Dimensional Linear regression over $l_q$ balls" by Garvesh Raskutti .et.al. In the right end of ...
0
votes
0answers
10 views

Autoregressive mixed effects generalised linear model for zero-inflated count data

I have a multi task learning scenario where I have $I$ items in each of $J$ groups and for each item $i$ I have $T_i$ observations $\{y_{i,j,t}\}_{t=1}^{T_i}$ (which are non-negative and mostly zero). ...
0
votes
0answers
37 views

Is max. Eigenvalue of k-sparse PCA always $\leq$ max. Eigenvalue of normal PCA on same dataset?

Is max. Eigenvalue of k-sparse PCA always less than or equal to the max. Eigenvalue of normal PCA on same dataset? K refers to the number of non zero eigenvalues when the dataset is of dimension n <...
1
vote
2answers
492 views

How to do dimension reduction with sparse data

I have 200 vectors representing the percentage marks for 200 different students in the different classes they took. The vectors are 22 dimensional (as there were 22 different classes in total) even ...
5
votes
1answer
99 views

Covering the unit sphere with sparse vectors

I'm looking for references for covering the $d$-dimensional unit sphere $$ \mathbb{S}^{d - 1} = \left\{ x \in \mathbb{R}^d : \| x \| = 1 \right\} $$ I'm trying to cover $\mathbb{S}^{d-1}$ with ...
1
vote
0answers
25 views

Correlation vs. logit coefficient

I am analysing a data set with a dichotomous independent and a dichotomous depenent variable. Persons r only is r = 0.047. However, if I use the same variables in a logistic regression the logit-...
2
votes
0answers
148 views

How to transform my sparse count data into normal distribution?

I am running glm on beetle counts data. My predictors are environmental variables and my response variable is the number of beetles. I ran three glms: The response variable $Y_1$ is the total number ...
2
votes
2answers
163 views

Database-friendly random projections with Numpy

In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
0
votes
0answers
30 views

Model selection consistency of Dantzig selector

Is it known that Dantzig selector of Candes and Tao: https://arxiv.org/abs/math/0506081 has model selection consistency, i.e, with high probability aporoaching 1 the model will select true features,(...
3
votes
2answers
449 views

Eligibility Traces vs Experience Replay

I am currently using the OpenAI Baselines implementation of DeepQ (paper found here). I am also utilizing Prioritized Experience Replay (paper found here). My problem involves sparse delayed rewards ...
0
votes
1answer
251 views

Cautions with sparse features [closed]

I got some sparse data for the first time and it's quite intimidating. After reading sklearn preprocessing docs it seems I should scale them with MaxAbsScaler (the sparseness is important). However, ...
0
votes
1answer
188 views

Does Matching Pursuit and Soft Thresholding return the same minimizer?

I wanted to understand if the solutions (minimizers) obtained by Matching Pursuit algorithms (say Basis Pursuit denoising) and Soft Thresholding yielded the same minimizer (same solution or same ...
1
vote
1answer
101 views

Does it make sense using Machine Learning techniques on a sparse features matrix?

I am trying to predict the sentiment (neg/neutral/pos) of a given text. To do so I use a LDA model (Latent Dirichlet Allocation) that is a topic discovery model. The LDA model works as follows: given ...
1
vote
1answer
130 views

sparse canonical correlation with PMA package in R - correlation coefficients

I'm new to canonical correlation analysis. I'm running a sparse canonical correlation analysis in R using the PMA package. My first question is why the correlation coefficients associated to the ...
4
votes
0answers
94 views

Why over-complete sparse basis?

In This Stanford Tutorial, it says "Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently... While techniques such as ...
1
vote
0answers
96 views

Unsupervised 1D clustering with sparse data

I am developing an unsupervised 1-dimensional clustering algorithm to detect regions of a protein in which genetic variants found in a population tend to concentrate. The analyzed data structure is a ...
1
vote
0answers
41 views

How to deal with infrequent features in a linear regression model?

I am working on a linear model problem, $y =f(X)$ where $X$ has around 200 columns and around 300K rows not surprisingly, I am using LASSO to bring down the complexity of the model. but when I ...
0
votes
1answer
194 views

How much sparsity is a problem for logistic regression?

I keep reading that sparsity (the number of cells with 0 observations in a cross tabulation of all variables in a model) is a problem for running logistic regression models because it biases odds ...
2
votes
1answer
476 views

Building Decision Tree on a high dimensional data set with sparse Boolean values

I have been trying to learn a decision tree on a data-set with almost 400 features. The target variable has only two values and the data is highly skewed towards the non-event class (90 % of the data ...
1
vote
1answer
92 views

Applying deep logistic regression on sparse labels

I have a dataset of 100 000 examples. Only 1% are positive (1000 examples). I want to predict the probability that a positive event happen. To do that I have built a 4 layer DNN (Linear W*x+b -> ReLU)...
0
votes
2answers
46 views

How to predict item purchase price with sparse purchase history

I want to predict the price of the next item a user purchases based on the prices of the items they have purchased in the past. The caveat is that most users have less than 3 previous purchases, so I ...
0
votes
0answers
66 views

What can be the reasons that L1-regularized NMF gets worse result than standard NMF in sparse matrix computation?

I apply L1-norm as a group sparsity constraint [1,2] into non-negative matrix factorization $V \approx WH$ for source separation. Objective functions: Standard NMF (Kullback-Leibler divergence): $...
1
vote
0answers
215 views

Regression - extremely skewed response with a large, sparse matrix of boolean predictors

I'm working with a dataset that contains: $y$: the response variable that is 98% zero, but in the remaining 2% of cases it has extremely skewed real values (not integers), ranging from sub 1 to over ...