A sparse matrix is a matrix where many of the elements are zeros.

learn more… | top users | synonyms

0
votes
0answers
16 views

Markov process with only $n$ most significant transition probabilities known

Suppose I want to simulate a Markov process with a discrete state space. Normally, I need to have all the transition probabilities known. However, in my situation, I can only measure the top $n$ most ...
3
votes
2answers
74 views

Theoretical justification for bag of words

Bag of words and visual bag of words are successful machine learning approaches. Does anyone know of a theoretical justification for why / when they work? What I am trying to explain is the good ...
0
votes
0answers
20 views

Regression - one input variable is mostly 0

Background: I have data obtained by sampling real-world, physical quantities. Say there are $10$ features. One of them, call it $x_1$, due to the nature of the sampling, is 0 most of the time. But ...
0
votes
0answers
8 views

what is the method in dictionary learning which does not have a overcomplete dictionary?

what is the method in dictionary learning which does not have a overcomplete dictionary? and what is the difference in minimization between these two methods (one using overcompelte dictionary and ...
0
votes
0answers
14 views

Using Proc distance with a very sparse matrix

I have a large sparse data set and I would like to apply segmentation of my customers. To give you an idea, I have more than 100 variables and 2.2 mln rows. Breakdown of my variables are as follows: ...
0
votes
0answers
24 views

Entropy weighted KMeans finishes after 1 Iteration. No Entropy in Data?

I want to cluster high dimensional sparse data (100k rows and 2k columns, 10-20 non-zero values per row). Each row represents a person and each column an attribute this person does or does not posess ...
1
vote
1answer
24 views

Best way to handle sparse + non-sparse data to create a model

I'm wondering what is the best way to handle sparse+non-sparse data in e.g. a Ridge regression using scikit learn. Ridge can handle both sparse and nonsparse data. Imagine something simple as a ...
0
votes
0answers
86 views

efficient “dot” product of two sparse vectors with uncertainty

I'm interested in extensions or analogs of the vector dot product that apply to sparse vectors in the case of uncertainty in the abscissa. The vectors I deal with are often of large (100,000 or ...
2
votes
1answer
378 views

Gradient boosting decision tree implementation

I am willing to implement my own GBM. I have been looking - unsuccessfully - for a clear article describing the implementation of gradient boosting machine for decision trees. Sources like this are ...
1
vote
0answers
35 views

Clustering algorithm advice for extracting key features in sparse data

I have the following dataset: consider a dataset $X$ of $1400 \times 600$. The rows represent households at time $1 \leq t \leq 14$. So I have $100$ households. The columns represent the programs that ...
2
votes
0answers
81 views

Miss Forest & Iterative PCA : How to handle very sparse matrix imputation?

I am currently benchmarking matrix completion methods (k-NN, RandomForest and iterative PCA) on multivariate normal data in which I introduce a certain proportion of NA (5 to 95%). My performance ...
0
votes
0answers
90 views

Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
7
votes
1answer
229 views

Is large scale PCA even possible?

The principal component analysis (PCA) algorithm assumes that columns of an input matrix have zero mean. This can be achieved easily, but when the input matrix is sparse, the centered matrix will now ...
2
votes
2answers
131 views

Finding pattern from sparse matrix

I have a large sparse matrix which represents whether the action is happened or not. Each columns represent each action. Each row represents time. The data is collected for every 15min. Zero in the ...
0
votes
0answers
18 views

Quantile or distribution estimation for continuous variable from sparse matrix

I'm not sure where to start and desperatley need help. I've got a somewhat sparse data set and I'm trying to do either a quantile estimation or a distribution estimation for one continuous variable. ...
1
vote
0answers
67 views

How to go from sparse matrix to linear regression model (using SVD)?

I am trying to replicate the Kosinski, Stillwell, & Graepel (2013) study about predicting private traits and attributes from Facebook like data for study purposes. First I have admit, however, ...
1
vote
0answers
67 views

Kernel K nearest neighbours with sparse data

I have a big sparse matrix (around 5 million of lines, 20 000 predictors), and I would like to run a kernelized k-NN on it. However, I don't know how to scale the data properly. So far, I have scaled ...
0
votes
0answers
17 views

Can I run an SVM on sparse temporal data without a regular time interval?

I have data of occurrences with timestamps that could be days or months apart. I'd like to enter the values natively as follows. Are there any SVM algorithms that can support such an input? day 1: ...
0
votes
0answers
30 views

Robust Sparse K Means clusters and valid index

I used the robust sparse k-means for clustering my dataset and I would like to calculate some distance-based statistics for evaluating my results. Should I compute them on the dissimilarity matrix ...
0
votes
0answers
8 views

Valid values for nvecs sparseDecom2

I hope this is the right place to ask, and that someone would know the answer I am searching for the optimal sparseness value to solve a sparse regression problem (using MRI data) that involve ...
1
vote
0answers
91 views

Feature scaling for non-negative sparse data

Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
0
votes
1answer
210 views

What's the best (Google chart) visualisation for displaying sparse timeline data across thousands of “columns”

I am trying to visualise a sparse dataset but am finding it hard to fit it into the standard categories of charts. I'm a developer building with Google Charts and I really want to stick with that ...
5
votes
1answer
123 views

Does the opposite of nested cross-validation make sense?

I'm asking the question from a machine learning point of view. I have a dataset with relatively high sparsity, so if I use nested cross-validation for my feature tuning and evaluation, that is tune ...
0
votes
0answers
25 views

commonalities and differences among groups in a matrix (based on features)

I have a (rather sparse) matrix with 7 columns and 66 rows. Each column represents a user group, each row represents a feature, and each cell represents a weighted value (see note below). I'd like to ...
0
votes
0answers
120 views

How to input sparse feature

In theano everything is symbolic, so how to input sparse feature in , for example, neural network? The setting is: the task is text application. the input is a mini-batch. Since theano sparse module ...
0
votes
0answers
25 views

Is there any reason that cross correlation would perform well or poorly on sparse binary arrays?

I am using matlabs xcorr to correlate simulated photon count data that has some Gaussian random noise set on top of it and it is working fine when the average value in the arrays is greater than one ...
0
votes
0answers
25 views

Minimum sample size required for sparse PCA

What is the minimum sample size that we need for filtering variables using sparse principal component analysis (sparse PCA, SPCA)?
2
votes
0answers
35 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
33 views

Sparsity of the matrix

We got a matrix of 500 users and 30 tracks. This matrix is complete full (it means every user rated explicit all 30 tracks). Every row is a combination of user id, music id and rating. Every user + ...
3
votes
0answers
100 views

sparse covariance/correlation thresholding

In our project, we would like to do some optimization on sparse matrices. The idea is to scrape massive amounts of data, form a covariance/correlation matrix, and form a sparsity pattern basically by ...
0
votes
1answer
41 views

What are the techniques to deal with classifying sparse categorical features?

Suppose I have a group of features each one is sparse with a few number of values (1-10) what are the required preprocessing steps required to avoid degradation of the performance of the classifier ...
0
votes
0answers
177 views

How can I run ordered logistic regression on a large sparse matrix in R

I have a sparse matrix X, 970283x9511, with 970283 documents and 9511 features. I have a vector y of length 970283 corresponding to a rating 1-5. I know of the glmnet package, which has binomial and ...
0
votes
0answers
123 views

Problems using pcr (from pls library) in R with large number of qualitative variables

I'm trying to classify a variable into either 0 or 1, using 50 factors, with a sample size of 2000. 25% of the dependent variables are 0 and the rest are 1. Of these factors, 30 are categorical. I've ...
0
votes
0answers
128 views

Handling sparse document term matrix

I am working with a corpus of several thousand documents (41,732) however the documents tend to be short (the median number of terms per document is 3) resulting in a sparse document term matrix. ...
2
votes
1answer
29 views

Spectral norm of a sparse Gaussian matrix

Suppose $G$ is an $m \times n$ matrix such that each entry of $G$ is a standard normal variable. We know that the spectral norm of $G$ scales as $\sqrt m + \sqrt n$. Now, given a set of indices $S$ ...
1
vote
0answers
32 views

Individual factor significance in multilevel sPLS-DA

I recently was asked by reviewers to "include p-values" with my multilevel sparse partial least squares analysis. In brief, I have a nested design with two factors, say treatment and sampling region. ...
2
votes
1answer
58 views

How to convert the objective function to canonic form of sparse coding?

As we know the conventional sparse coding problem (LASSO) is: $\min_{\alpha} \| X-D\alpha\|_F^2 + \lambda \|\alpha\|_{1} \tag{1}$ where $X$ , $D$, and $\alpha$ are data, dictionary and coefficients ...
3
votes
0answers
100 views

Generate Symmetric Positive Definite Matrix with a pre-specified Sparsity

I am trying to generate a correlation matrix $p\times p$ (symmetric p.s.d) with a pre-specified sparsity structure (specified by a graph on $p$ nodes). The nodes that are connected in the graph have ...
1
vote
1answer
113 views

Calculating means and confidence intervals when most data points are 0

I am looking at data set that has four groups. In each group, the data is mostly, 99+% of time, composed of zeros, but, when it is not zero it can be any float number (e.g., 0.01 to 921.2, with most ...
1
vote
2answers
347 views

How to build a predictive model with a billion of sparse features?

I am making a model to learn a dataset which has a big feature number and sparse samples (I am planning to use logistic regression). The feature number can be as big as 1,000,000,000. It is sparse ...
0
votes
2answers
366 views

Kernel SVM on sparse data

I have a sparse dataset where a lot of the columns (features) contain mostly zero values. Class labels are multiple discrete categories (10 classes to be precise). I'm wondering if this should trouble ...
4
votes
1answer
240 views

Logistic regression with sparse predictor variables

I am currently modeling some data using a binary logistic regression. The dependent variable has a good number of positive cases and negative cases - it is not sparse. I also have a large training set ...
2
votes
0answers
142 views

What guidelines should be followed for using Neural Networks with sparse inputs

I have extremely sparse inputs, e.g. locations of certain features in an input image. Further each feature can have multiple detections (not sure if this will have a bearing on the design of the ...
4
votes
1answer
710 views

Cross correlation for very sparse binary data

I have a very large (5271159x60) sparse (~2.5%) binary matrix, and I'd like to calculate the cross correlation between each of the columns (sensors) for a series of lags from -10:10, which would give ...
3
votes
1answer
132 views

Euclidean distance with sparse and high dimension data

I have texts for a bunch of objects. From each text, I removed the stop words, and took each word as an attribute of the object. I then gave each word a rating based on sentiment analysis, so that the ...
1
vote
0answers
133 views

Reduction of sparse features for machine learning

I'm trying to use a 1D histogram as a feature for machine learning. A histogram instance can be very sparse and the range of its bins is theoretically unbounded. Moreover it is expected that non-zero ...
0
votes
1answer
153 views

Tanh activation function and sparsity constraint

According to Lecun's paper "effient backprop" [1] the tanh activation function should be preferred over the logistic activation function for the hidden units in neural networks. For the tanh units ...
2
votes
0answers
188 views

The name data sparsity in different applications

I am recently surveying the techniques or algorithms which handle the data sparsity problems in various fields. And I find quite similar name "data sparsity" or "sparse data" is used including the ...
0
votes
1answer
36 views

How to do if the most training data is sparse

Consider a problem like this You have a customer profiling application(say classic telecoms data) You have customer transactions(lots of them) you want to find rules There is a data element which is ...
1
vote
0answers
43 views

Using an odds ratio when data is sparse

Suppose I have around 20 exposures that potentially affect an outcome and I want to see which exposures have bigger impacts on the outcome. So I want to calculate each exposures' odds ratios by ...