Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sparse]

A sparse matrix is a matrix where many of the elements are zeros. The tag can also be used for sparsity in other contexts, such as regression models with sparsity, or the "bet on sparsity"-principle.

0
votes
0answers
8 views

How much data is considered “sparse” for fitting a mixed (Beta Geometric) distribution with 4 shape parameters?

I'm using CamDavidsonPhillips Customer Lifetime Value library to calculate CLV, and it uses a distribution based on Peter Fader's work on the subject that fits a Gamma distribution to model customer ...
5
votes
1answer
61 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...
1
vote
0answers
26 views

Random Forest Regression with sparse data in Python

I am working on a Random Forest regression model to predict housing prices. I have about 500k rows of data with the following information: 1.House area in square meters. 2.Number of rooms. 3.City. ...
0
votes
0answers
8 views

Sparse solutions: linear systems vs logistic regression

It is known in the field of compressed sensing/sparse approximations that if $$Ax = b$$ has sparse solution (with $s$ nonzeros), then there is a condition which states that it is unique, if $$s \leq 0....
0
votes
0answers
23 views

Sparse coding and feature learning

Recently I tried to understand sparse coding and its application to classification. But there is no way to check whether I understand correctly, so I have a few questions about this algorithm. I ...
0
votes
0answers
16 views

How to develop features for deep learning from cart items data?

I wonder how to approach building set of features to feed deep learning model (eg convnet) from cart items data: 5pcs of product1 1pcs of product5 2pcs of product8 Assuming 30-50 products per ...
1
vote
0answers
32 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
1
vote
1answer
23 views

finding sparse regions in time series data

I have several hundred years of church baptisms that will be searched by people wanting to find the baptisms of their ancestors. I want to call attention to periods in the records in which the number ...
2
votes
1answer
285 views

Do you standardize the data before PCA whitening?

I have a data set ranged in different scales as well as some variables are sparse, for example, ...
0
votes
0answers
16 views

Normalizing sparse matrix by mean, should the mean be calculated excluding zero?

I have very sparse matrix (70% sparsity) which I want to normalize by mean. I tried using mean both include and exclude zero. The histogram between count (y-axis) and value (x-axis) shows The value ...
1
vote
0answers
77 views

GAN and NN for sparse data

I have a set of images which represent some correlated sparse data $x_1,\ldots ,x_n$. there are a number of specific pixels in the images which might hold value or not (with some probability), while ...
1
vote
1answer
97 views

Simulation of low rank and sparse matrix

I am having trouble simulating a matrix which is low rank and sparse (sparse along both rows and columns). One way to simulate a low-rank matrix is by generating a random matrix, then taking SVD and ...
4
votes
1answer
565 views

Can it be over fitting when validation loss and validation accuracy is both increasing?

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, ...
2
votes
1answer
54 views

Amount and sparsity of data for recommender systems

I'm starting to work in a project that will have a recommender system as one of its components. I'm trying to figure out if I have the right type of data for the recommender. The data contains ...
0
votes
0answers
11 views

What kind of sparse matrix representation is this?

I am putting together a wrapper for a quadratic programming library. I am going through the C example here but I don't understand the indexing used for the matrices. The relevant excerpt is below, ...
1
vote
0answers
44 views

Confidence region for multinomial distribution with k=7 - including 5 zero values in dataset

I have a multinomial distribution with $k=7$ and an observed dataset $n_i = \{62, 35, 0, 0, 0, 0, 0\}$. While it was quite expected that $n_i = 0$ for $i \in \{3,4,...,7\}$, there was no way to ...
0
votes
0answers
81 views

Getting explained variance from Scikit's SparsePCA model?

I am trying to apply SparsePCA by Scikit: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.SparsePCA.html#sklearn.decomposition.SparsePCA But I don't know how to extract the ...
0
votes
0answers
27 views

autoencoder parameters effect matlab

i'm new in autoencoders and matlab i'm applying this tutorial https://www.mathworks.com/help/nnet/ug/construct-deep-network-using-autoencoders.html after extrating the features i'm trying to check ...
3
votes
0answers
19 views

How to prove oracle properties in Fan and Li (2001) paper

I am studying Fan and Li's 2001 paper "Variable selection via nonconcave penalized Likelihood andits oracle properties" but I am having troubles understanding Theorem 1 proof (page 1359). I follow the ...
0
votes
0answers
28 views

Can Chi-Sq Test be used to explore the independence of categorical variables on response variable?

I am investigating the independence of some categorical variables (apps, campaigns...) on the response variable (click through rate). I have some doubts on using Chi-Sq Test here: Chi-Sq Test can be ...
1
vote
1answer
50 views

Data sparsity becomes a problem

I read this in a paper Developing an approach toward virtual synthesis parameter screening introduces two primary computational challenges: data sparsity and data scarcity. ...Such canonical ...
0
votes
1answer
13 views

Question about minimax risk sparse sparse condition

I am studying statistical leaning theory. Especially the paper "Minimax rates of Estimation for High Dimensional Linear regression over $l_q$ balls" by Garvesh Raskutti .et.al. In the right end of ...
0
votes
0answers
9 views

Autoregressive mixed effects generalised linear model for zero-inflated count data

I have a multi task learning scenario where I have $I$ items in each of $J$ groups and for each item $i$ I have $T_i$ observations $\{y_{i,j,t}\}_{t=1}^{T_i}$ (which are non-negative and mostly zero). ...
0
votes
0answers
34 views

Is max. Eigenvalue of k-sparse PCA always $\leq$ max. Eigenvalue of normal PCA on same dataset?

Is max. Eigenvalue of k-sparse PCA always less than or equal to the max. Eigenvalue of normal PCA on same dataset? K refers to the number of non zero eigenvalues when the dataset is of dimension n <...
1
vote
2answers
256 views

How to do dimension reduction with sparse data

I have 200 vectors representing the percentage marks for 200 different students in the different classes they took. The vectors are 22 dimensional (as there were 22 different classes in total) even ...
5
votes
1answer
88 views

Covering the unit sphere with sparse vectors

I'm looking for references for covering the $d$-dimensional unit sphere $$ \mathbb{S}^{d - 1} = \left\{ x \in \mathbb{R}^d : \| x \| = 1 \right\} $$ I'm trying to cover $\mathbb{S}^{d-1}$ with ...
1
vote
0answers
22 views

Correlation vs. logit coefficient

I am analysing a data set with a dichotomous independent and a dichotomous depenent variable. Persons r only is r = 0.047. However, if I use the same variables in a logistic regression the logit-...
2
votes
0answers
97 views

How to transform my sparse count data into normal distribution?

I am running glm on beetle counts data. My predictors are environmental variables and my response variable is the number of beetles. I ran three glms: The response variable $Y_1$ is the total number ...
1
vote
2answers
151 views

Database-friendly random projections with Numpy

In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
0
votes
0answers
24 views

Model selection consistency of Dantzig selector

Is it known that Dantzig selector of Candes and Tao: https://arxiv.org/abs/math/0506081 has model selection consistency, i.e, with high probability aporoaching 1 the model will select true features,(...
2
votes
2answers
295 views

Eligibility Traces vs Experience Replay

I am currently using the OpenAI Baselines implementation of DeepQ (paper found here). I am also utilizing Prioritized Experience Replay (paper found here). My problem involves sparse delayed rewards ...
0
votes
1answer
189 views

Cautions with sparse features [closed]

I got some sparse data for the first time and it's quite intimidating. After reading sklearn preprocessing docs it seems I should scale them with MaxAbsScaler (the sparseness is important). However, ...
0
votes
1answer
147 views

Does Matching Pursuit and Soft Thresholding return the same minimizer?

I wanted to understand if the solutions (minimizers) obtained by Matching Pursuit algorithms (say Basis Pursuit denoising) and Soft Thresholding yielded the same minimizer (same solution or same ...
1
vote
1answer
70 views

Does it make sense using Machine Learning techniques on a sparse features matrix?

I am trying to predict the sentiment (neg/neutral/pos) of a given text. To do so I use a LDA model (Latent Dirichlet Allocation) that is a topic discovery model. The LDA model works as follows: given ...
1
vote
0answers
105 views

sparse canonical correlation with PMA package in R - correlation coefficients

I'm new to canonical correlation analysis. I'm running a sparse canonical correlation analysis in R using the PMA package. My first question is why the correlation coefficients associated to the ...
0
votes
0answers
101 views

how to deal with the sparsity of bag of words when using them for classification or other tasks?

I am trying to perform sentiment analysis. There are only 12500 vectors (BOW) but the vocabulary is huge and each vector is very high dimensional (70k). So loading this into memory is a problem on a ...
4
votes
0answers
89 views

Why over-complete sparse basis?

In This Stanford Tutorial, it says "Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently... While techniques such as ...
1
vote
0answers
85 views

Unsupervised 1D clustering with sparse data

I am developing an unsupervised 1-dimensional clustering algorithm to detect regions of a protein in which genetic variants found in a population tend to concentrate. The analyzed data structure is a ...
1
vote
0answers
37 views

How to deal with infrequent features in a linear regression model?

I am working on a linear model problem, $y =f(X)$ where $X$ has around 200 columns and around 300K rows not surprisingly, I am using LASSO to bring down the complexity of the model. but when I ...
0
votes
1answer
110 views

How much sparsity is a problem for logistic regression?

I keep reading that sparsity (the number of cells with 0 observations in a cross tabulation of all variables in a model) is a problem for running logistic regression models because it biases odds ...
1
vote
1answer
362 views

Building Decision Tree on a high dimensional data set with sparse Boolean values

I have been trying to learn a decision tree on a data-set with almost 400 features. The target variable has only two values and the data is highly skewed towards the non-event class (90 % of the data ...
1
vote
1answer
83 views

Applying deep logistic regression on sparse labels

I have a dataset of 100 000 examples. Only 1% are positive (1000 examples). I want to predict the probability that a positive event happen. To do that I have built a 4 layer DNN (Linear W*x+b -> ReLU)...
0
votes
2answers
43 views

How to predict item purchase price with sparse purchase history

I want to predict the price of the next item a user purchases based on the prices of the items they have purchased in the past. The caveat is that most users have less than 3 previous purchases, so I ...
0
votes
0answers
59 views

What can be the reasons that L1-regularized NMF gets worse result than standard NMF in sparse matrix computation?

I apply L1-norm as a group sparsity constraint [1,2] into non-negative matrix factorization $V \approx WH$ for source separation. Objective functions: Standard NMF (Kullback-Leibler divergence): $...
1
vote
0answers
172 views

Regression - extremely skewed response with a large, sparse matrix of boolean predictors

I'm working with a dataset that contains: $y$: the response variable that is 98% zero, but in the remaining 2% of cases it has extremely skewed real values (not integers), ranging from sub 1 to over ...
2
votes
2answers
3k views

difference between convex and concave functions

what is the difference between convex, non-convex, concave and non-concave functions? how will we come to know that the given function is convex or non-convex? and if a function is non-convex then it ...
4
votes
0answers
25 views

using sparse models to accelerate training

I was going through a tutorial that highlights the importance of using sparse models to get "lightning fast models" i.e. to accelerate the training process. What are the issues in using a sparse ...
0
votes
1answer
172 views

Many sparse and many dense features in Random Forest

I have a large dataset (approx. 400,000 rows, 650 columns). Out of this 650 columns, 500 are dense (nonzero over 10%) and 150 are sparse (nonzero less than 10%). As expected, the random forest has ...
1
vote
1answer
49 views

$\ell_0$ penalised and $K$ sparse problems

Consider the $\ell_0$ penalized problem: $$\min_{x\in \mathbb{R}^n} \frac{1}{2}\|Ax-b\|_2^2+\lambda\|x\|_0 \qquad \qquad \qquad \qquad \qquad \qquad (1)$$ and $K$-sparse problem $$\min\frac{1}{2}\|Ax-...
0
votes
2answers
871 views

Weather data in time series predictions

Disclaimer: I know this is a long-ish post but I don't need code solutions just high level general direction approaches that are usually used in situations like these. So let's say I want to predict ...