A sparse matrix is a matrix where many of the elements are zeros.

learn more… | top users | synonyms

0
votes
0answers
16 views

Is there any reason that cross correlation would perform well or poorly on sparse binary arrays?

I am using matlabs xcorr to correlate simulated photon count data that has some Gaussian random noise set on top of it and it is working fine when the average value in the arrays is greater than one ...
0
votes
0answers
10 views

Minimum sample size required for sparse PCA

What is the minimum sample size that we need for filtering variables using sparse principal component analysis (sparse PCA, SPCA)?
1
vote
0answers
23 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
20 views

Sparsity of the matrix

We got a matrix of 500 users and 30 tracks. This matrix is complete full (it means every user rated explicit all 30 tracks). Every row is a combination of user id, music id and rating. Every user + ...
3
votes
0answers
77 views

sparse covariance/correlation thresholding

In our project, we would like to do some optimization on sparse matrices. The idea is to scrape massive amounts of data, form a covariance/correlation matrix, and form a sparsity pattern basically by ...
0
votes
1answer
31 views

What are the techniques to deal with classifying sparse categorical features?

Suppose I have a group of features each one is sparse with a few number of values (1-10) what are the required preprocessing steps required to avoid degradation of the performance of the classifier ...
0
votes
0answers
45 views

How can I run ordered logistic regression on a large sparse matrix in R

I have a sparse matrix X, 970283x9511, with 970283 documents and 9511 features. I have a vector y of length 970283 corresponding to a rating 1-5. I know of the glmnet package, which has binomial and ...
0
votes
0answers
42 views

Problems using pcr (from pls library) in R with large number of qualitative variables

I'm trying to classify a variable into either 0 or 1, using 50 factors, with a sample size of 2000. 25% of the dependent variables are 0 and the rest are 1. Of these factors, 30 are categorical. I've ...
0
votes
0answers
44 views

Handling sparse document term matrix

I am working with a corpus of several thousand documents (41,732) however the documents tend to be short (the median number of terms per document is 3) resulting in a sparse document term matrix. ...
2
votes
1answer
18 views

Spectral norm of a sparse Gaussian matrix

Suppose $G$ is an $m \times n$ matrix such that each entry of $G$ is a standard normal variable. We know that the spectral norm of $G$ scales as $\sqrt m + \sqrt n$. Now, given a set of indices $S$ ...
0
votes
0answers
12 views

Preprocessing for dictionary learning

What are the pre-processing steps on data before dictionary learning,say using KSVD? Do we have to standardize the data first? Thanks.
1
vote
0answers
17 views

Individual factor significance in multilevel sPLS-DA

I recently was asked by reviewers to "include p-values" with my multilevel sparse partial least squares analysis. In brief, I have a nested design with two factors, say treatment and sampling region. ...
2
votes
1answer
40 views

How to convert the objective function to canonic form of sparse coding?

As we know the conventional sparse coding problem (LASSO) is: $\min_{\alpha} \| X-D\alpha\|_F^2 + \lambda \|\alpha\|_{1} \tag{1}$ where $X$ , $D$, and $\alpha$ are data, dictionary and coefficients ...
3
votes
0answers
75 views

Generate Symmetric Positive Definite Matrix with a pre-specified Sparsity

I am trying to generate a correlation matrix $p\times p$ (symmetric p.s.d) with a pre-specified sparsity structure (specified by a graph on $p$ nodes). The nodes that are connected in the graph have ...
1
vote
1answer
75 views

Calculating means and confidence intervals when most data points are 0

I am looking at data set that has four groups. In each group, the data is mostly, 99+% of time, composed of zeros, but, when it is not zero it can be any float number (e.g., 0.01 to 921.2, with most ...
1
vote
2answers
128 views

How to build a predictive model with a billion of sparse features?

I am making a model to learn a dataset which has a big feature number and sparse samples (I am planning to use logistic regression). The feature number can be as big as 1,000,000,000. It is sparse ...
0
votes
2answers
80 views

Kernel SVM on sparse data

I have a sparse dataset where a lot of the columns (features) contain mostly zero values. Class labels are multiple discrete categories (10 classes to be precise). I'm wondering if this should trouble ...
1
vote
0answers
58 views

logistic regression with sparse predictor variables

I am currently modeling some data using a binary logistic regression. The dependent variable has a good number of positive cases and negative cases - it is not sparse. I also have a large training set ...
0
votes
0answers
23 views

Small sample size : dealing with bootstraping for linear or nonlinear multiple regression

I am wondering to heal my ignorance from your experiences or your modeling knowledge. I have many matrices of quantitative variables, let me start with three matrices of proportions.To express ...
0
votes
0answers
30 views

Sparse PLS: algorithm for variable selection and model fitting

In the spls package in R (based on the manuscript by Chun and KeleĊŸ [1]), there is a separate specification for the variable selection and fitting in the main function, ...
0
votes
0answers
22 views

What guidelines should be followed for using Neural Networks with sparse inputs

I have extremely sparse inputs, e.g. locations of certain features in an input image. Further each feature can have multiple detections (not sure if this will have a bearing on the design of the ...
3
votes
1answer
224 views

Cross correlation for very sparse binary data

I have a very large (5271159x60) sparse (~2.5%) binary matrix, and I'd like to calculate the cross correlation between each of the columns (sensors) for a series of lags from -10:10, which would give ...
0
votes
0answers
18 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
3
votes
1answer
88 views

Euclidean distance with sparse and high dimension data

I have texts for a bunch of objects. From each text, I removed the stop words, and took each word as an attribute of the object. I then gave each word a rating based on sentiment analysis, so that the ...
1
vote
0answers
39 views

Reduction of sparse features for machine learning

I'm trying to use a 1D histogram as a feature for machine learning. A histogram instance can be very sparse and the range of its bins is theoretically unbounded. Moreover it is expected that non-zero ...
0
votes
1answer
65 views

Tanh activation function and sparsity constraint

According to Lecun's paper "effient backprop" [1] the tanh activation function should be preferred over the logistic activation function for the hidden units in neural networks. For the tanh units ...
1
vote
0answers
46 views

The name data sparsity in different applications

I am recently surveying the techniques or algorithms which handle the data sparsity problems in various fields. And I find quite similar name "data sparsity" or "sparse data" is used including the ...
0
votes
0answers
16 views

How to do if the most training data is sparse

Consider a problem like this You have a customer profiling application(say classic telecoms data) You have customer transactions(lots of them) you want to find rules There is a data element which is ...
1
vote
0answers
30 views

Using an odds ratio when data is sparse

Suppose I have around 20 exposures that potentially affect an outcome and I want to see which exposures have bigger impacts on the outcome. So I want to calculate each exposures' odds ratios by ...
0
votes
1answer
22 views

Utilizing A Correlation Matrix Derived from a Sparse Matrix

I have large correlation matrix in Excel that I'd like to use to inform my choice of explanatory variables in a multiple linear regression model. One problem is that the initial data was very sparse, ...
0
votes
1answer
157 views

LibFM sparse data format

Does the order of columns in the the sparse format matters in case of libfm ? Can I list the non zero components of X in libfm in any order in a row.
0
votes
0answers
88 views

Sparse ELM vs SVM

What's the difference between SVM and Sparse Extreme Learning Machine with Gaussian kernel proposed in the following paper:http://www.ntu.edu.sg/home/egbhuang/pdf/Sparse-ELM-IEEE-T-Cybernetics.pdf As ...
0
votes
1answer
42 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...
2
votes
1answer
420 views

Clustering algorithms for extremely sparse data

I am trying to cluster an extremely sparse text corpus, and I know the number of clusters (my data is the title and author list of scientific publications, for which I already know the number of ...
1
vote
0answers
109 views

regression algorithms which work with sparse categorical predictors

I am working with a very sparse problem with a large number of categories per feature and I am currently looking for existing machine learning regression algorithm implementations which can either ...
1
vote
2answers
374 views

Algorithm and R implementation of sparse PCA

Does anyone know where I can find an algorithm, as well as an R implementation of it, to carry out sparse principal component analysis (PCA)?
0
votes
0answers
31 views

Heavy tailed distribution results in sparse random variable?

I often come across that, in Bayesian inference, to encourage sparseness, we should choose a heavy-tailed probability distribution as prior. AFAIK, in heavy-tailed distribution, it's more likely for ...
1
vote
1answer
51 views

Statistical Practices Using Sparse Data: Methods for Approximating Standard Deviation

Suppose I know that for a discrete, non-negative r.v. $X$ that $X | X \geq 1$ has $\mu = 3.3$ while when $X \geq 0$ has $\mu = 2.1$. That is, the subset of the population that already has a value ...
4
votes
0answers
191 views

Compute Shannon entropy between every row of a large, sparse matrix

I have a sparse, binary matrix of user (rows) and items (columns). Each element of this matrix is either 0 or 1: ...
3
votes
1answer
262 views

Machine learning techniques for spam detection, and in general for text classification

I am going to configure a system for spam detection. What I have is a dataset of labeled (spam/not-spam) strings containing, mostly, sentences. I have a background in machine learning techniques, but ...
7
votes
2answers
202 views

Selecting the number of sparse principal components to include in regression

Does anyone have experience with approaches for selecting the number of sparse principal components to include in a regression model?
1
vote
0answers
78 views

Clustering time-shifted sales time-series

I need to perform clustering and classification of time series of weekly sales of different products. My data are weekly sales of different products in differest areas (stores). The challenges on this ...
2
votes
0answers
44 views

Random classification forests for extremely sparse response variables

I have a response variable that can be $A,B,C$. It is very sparse, meaning 99% of the sample is $B$ and the rest is approximately evenly divided between $A$ and $C$. How do I predict this variable in ...
0
votes
1answer
159 views

Sparse PCA/Dictionary learning when the features are extremely sparse?

I am trying to do sparse PCA/dictionary learning, that is decompose a matrix $X\approx UV$ where the loading matrix $V$ is sparse, usually enforced with an $\ell_1$ penalty (the difference between ...
4
votes
1answer
2k views

Clustering algorithms that operate on sparse data matricies [closed]

I'm trying to compile a list of clustering algorithms that are: Implemented in R Operate on sparse data matrices (not (dis)similarity matrices), such as those created by the sparseMatrix function. ...
4
votes
2answers
207 views

How exactly is sparse PCA better than PCA?

I learnt about PCA a few lectures ago in class and by digging more about this fascinating concept, I got to know about sparse PCA. I wanted to ask, if I'm not wrong this is what sparse PCA is: In ...
0
votes
2answers
211 views

R or python implementation of sparse PCA for p>n

According to this paper, there are 2 algorithms to perform sparse PCA. One is better if $p>n$. I need to run SPCA on a $2000\times12000$ matrix so I am looking for an implementation of this ...
2
votes
1answer
71 views

Interpretation of lasso recovery results

When people say that lasso regression can under certain assumptions recover "the support", i.e. non-zero regression weights, what does this mean? This cannot mean causal recovery, because Pearl has ...
0
votes
1answer
168 views

How Can I use some variables selected by LASSO?

I am very new about statistics. So, please understand if my question is somewhat awkward, and please give me related any advice. I have some data set. X = 500 x 100 (500 observations x 100 ...
2
votes
1answer
176 views

Limitations of Cohen's kappa for sparse data?

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, ...