Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
9 views

Gaussian Mixture Model with Custom Distance Metric

I have some 1D data that I want to cluster using Mixture of Gaussian. However, the data "wraps around" at two extremes. Specifically, I have a list of angles from $-\pi$ to $\pi$ and the data near two ...
0
votes
0answers
25 views

Salesforce data mining insight using Python [on hold]

I am working with data in a multinational corporation and I need some inspiration on how our data can be brought to life through: External data sources (publicly accessible) Using Python ...
0
votes
0answers
16 views

Penalized ML estimation of non-linear probit

I have a model of the form $P(y_i=1) = \Phi(\frac{w_1^{\beta'x_i}-w_2^{\beta'x_i}}{\sigma' x_i})$ where $y_i$ is a binary response, $\Phi$ is the normal CDF, $w_1$ and $w_2$ are non-negative ...
0
votes
0answers
20 views

PyMC consistently under estimating results found in paper. Possibly not sampling enough?

I have been trying to build confidence in (my ability to correctly use) PyMC by working examples. Namely, I have been working on Chickering and Pearl 1997, and more specifically on their 'artificial' ...
0
votes
0answers
25 views

what's the best empirical macro/micro F1 score?

Theoretically it should be 1. In the following presentation it's said that "0.5 to 0.55 (micro) F1 score is tops for multi-label classification problems" I tried to investigate this statement but ...
0
votes
0answers
5 views

How Are Integers Stored In Excel [migrated]

I wrote a python script that uses the CSV and xlrd packages to convert a .xlsx binary excel file to a tab delimited .txt file. Unfortunately, every cell is converted to a string (which is ...
2
votes
2answers
37 views

Clustering a correlation matrix

I have a correlation matrix which states how every item is correlated to the other item. Hence for a N items, I already have a N*N correlation matrix. Using this correlation matrix how do I cluster ...
0
votes
0answers
7 views

What is the fastest way to get the document term matrix in scikit learn?

I am using scikit learn CountVectorizer on top of ~11K documents, each of size ~5000 words. It takes ~ 1 hour to generate the tdm (document term matrix). Is there a much faster way to generate the ...
0
votes
0answers
3 views

Hard Clustering using GAAC [migrated]

I am trying to implement hard clustering using NLTK gaac.py . So far I have the code . I below. I am trying to cluster around 7000 text documents . Each text file has a single post from a forum ...
0
votes
0answers
39 views

Kalman filter with control inputs in python?

i am trying to fit a simple kalman filter with input controls (in this case step input) in python. i am using filterpy (http://filterpy.readthedocs.org/). my code is: ...
0
votes
0answers
25 views

Why does Support Vector Regression slow down after several iterations?

TLDR: Why does SVR slow down on my machine after multiple runs in IPython/sklearn? I'm trying to grid-search to optimize some parameters in Support Vector Regression with a problem that has 2 ...
0
votes
0answers
15 views

Obatining PCA residusals in Python's scikit-learn [migrated]

I'm using scikit-learn to conduct PCA on a large dataset with the goal of removing large, common sources of variance from a matrix X. Thus, I'd like to produce a ...
0
votes
1answer
31 views

Least-square fit with uneven distribution of data

I'd like to perform least-squares fit to data which is unevenly distributed on the x-axis. For example, if I was to bin the data, it would be something like x = 0~5: 10 data points x = 5~10: 20 ...
0
votes
0answers
12 views

How to find the probalities of a,b,c (mutually exclusive events) when i turn the problem into three binary classification problems?

I am using scikits linear logisitc regression to classify three events a,b and c. it works better (score) when i convert them into a binary classification model. such as: 1. M1 classifies a or b 2. ...
0
votes
0answers
4 views

Prepare data for scikit-learn [migrated]

I am working on a small NLP project of authorship attribution: I have some texts from two authors and I want to say who wrote them. I have some pre-processed text (tokenized, pos-tagged, ect.) and I ...
0
votes
0answers
23 views

Latent Semantic Analysis with automatically discovered priors - gensim

I have around 20,000 English words in a set, mostly nouns. Something like {berry,bloom,buddy,comic,front,mind,charlie,consultants,destination,enterprise,experts,lady,stores,weight,arms,autism,balloon} ...
0
votes
1answer
57 views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
3
votes
1answer
31 views

Likeness of brands using tweets - is Chi-square appropriate?

I'm trying to determine if brand x is more similar to brand y or brand z using tweets. I have a data set for the words that occurred in each brand's tweets, and I'm considering using the chi-square ...
3
votes
0answers
44 views

State of the Art versions of Generalized Additive Models

Generalized Additive Models [Tribshirani 86] was well received with over 1335 Citations. I am also aware of the popular version of GAM - the Multivariate Adaptive Regression Splines [MARS by Friedman ...
4
votes
2answers
62 views

Is the chi-squared test appropriate with many small counts in a 5x2 table?

I have two sample populations, A, and B, which are independent. ...
0
votes
1answer
32 views

Weekly data normalization - Python

I have a weekly dataset and I have to normalize this data. Data is something like this : ...
0
votes
1answer
41 views

Questions about weather prediction in scikit learn

Hello I am a high school student doing research on weather. I have a dataset that has four columns each labeled with time, pressure, and lat/long. I am confused on the cross validation process. What ...
0
votes
0answers
27 views

How to do ranking with scikit-learn random forest model

I have a training dataset that I've developed, that has the following format: ------------------------------ | User ID | Item | Label | ------------------------------ | 001 | umbrella | 0 ...
0
votes
1answer
35 views

Log posterior function in PYMC

my question concerns the logp function in the PYMC package in Python. Ultimately I want to calculate a quantity that goes by many names, namely the Bayes-factor/ evidence/ marginal-likelihood of the ...
1
vote
0answers
63 views

Prediction based on multiple time series - Python

I have 3 predictors and 1 variable that represents ground truth. They all are linked time series. My purpose is with the 3 predictors to try to forecast the ground truth data. For example : ...
-1
votes
1answer
56 views

Prediction on multiple regression - Python

I have 3 list of value and 1 ground truth data. They all belongs to the same time series. My purpose is with 3 list try to forecast the ground truth data. For example : ...
1
vote
1answer
58 views

Simple Multivariate Bayesian Method in Python

I am trying to follow the Bayesian method described in this text. The python notebook goes through the example of creating two Poisson functions describing a change in SMS frequency at some point tau. ...
0
votes
0answers
46 views

MonteCarlo simulations to test light curve variability

I have an average orbital light curve for a source, that is, binned count rate vs orbital phase, where the count rate are averaged over a number of orbit. I want to run MonteCarlo simulations to find ...
0
votes
1answer
41 views

Python - SkLearn Logistic Regression: One-by-one train instance

Here is my question, I have a huge train set so I can't load it in memory and apply this code. ...
1
vote
2answers
149 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
0
votes
1answer
30 views

Regression with a kernel

I have a fixed kernel and a set of points. I do SVC with the flavor of SVM classification i'm working on (assume it's just a regular SVM) and i obtain a classifier represented by an explicit vector of ...
1
vote
0answers
17 views

Sampling subset to span entire range of full set (*not* to be representative), in order to construct some sort of lookup table

I have a large number of $N$ (20770) measurements. I need to perfom a calculation on all of them, but this is computationally too expensive. Therefore, I am looking for a way to select a subset of $p$ ...
0
votes
0answers
26 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
0answers
41 views

Pre-processing before digit recognition for NN & CNN trained with MNIST dataset

I'm trying to classify handwriting digits, written by myself and a few friends, by usign NN and CNN. In order to train the NN, MNIST dataset is used. The problem is the NN trained with MNIST dataset ...
0
votes
0answers
24 views

Custom Impulse Response Functions for VAR

I am working on a VAR model for the Safe Assets in the U.S. economy. The demand for Safe Assets has increased as needs to pledge them as collateral, or the response after the '08 crisis has been risk ...
1
vote
1answer
54 views

Marginal Likelihood in PYMC

I am using the PYMC toolbox in python in order to carry out a model selection problem using MCMC. What I would like to have for each model is the marginal log-likelihood (i.e. model evidence). The ...
0
votes
0answers
18 views

Repeated utility values in Value Iteration (Markov Decision Process)

I am trying to implement the value iteration algorithm of the Markov Decision Process using python. I have one implementation. But, this is giving me many repeated values for the utilities. My ...
2
votes
0answers
108 views

A smarter way for evaluating combinations of samples to optimize an overall score

I have the following problem (which I simplified for clarity) that I want to find k out of n samples (here: rows) that yield the maximum score (sum of the values in ...
4
votes
1answer
111 views

Review of Box-Jenkins methodology

i just finished developing an ARMAX model with python (mostly statsmodels) in order to forecast some data. My next step is to test the data (24 time series) with the given ARMAX model. As i need to ...
1
vote
1answer
34 views

Automatic selection of lowest information criterion comes with warning

I am building a forecasting model (ARMA) and found the very useful code-object arma_order_select_ic(see code below). It all works, however, each calculation comes ...
2
votes
1answer
76 views

Differences in Spearman coefficient between R and pandas

I've noticed a small difference between pandas and R with regards to how they calculate Spearman coefficients. It seems as if some rounding occurs. I see no such difference when calculating Kendall or ...
1
vote
0answers
24 views

RandomForest - why isn't it predicting well with manually-selected test sets?

I am using python sklearn.ensemble to do a RandomForestClassifier on about 800K rows of data, coupled with sklearn.cross_validation to generate the train/test sets. When it completes, it says on the ...
2
votes
0answers
14 views

False Positives in Real Time Classification

I am doing a sliding window binary classification. I have time series data and I am running a time window over this data and let a classifier produce a decision probability. Based on this probability, ...
3
votes
1answer
64 views

Is it possible to seed RANSAC with a given line?

I am analyzing a stream of data and I want to seed every new instance with the best guess output (line) of the previous, so as to eventually converge. Given that Scikit Learn - RANSAC is an iterative ...
3
votes
2answers
111 views

Visualization of random distribution with 3 variables

Suppose I have a joint distribution of three random variables $x,y,z$, $P(x,y,z)$. For simplicity, let's suppose those three rvs. are discrete. The distribution will be represented in Python as a ...
0
votes
1answer
27 views

sklearn.tree.export_graphviz values do not add up to samples

When I run tree.export_graphviz() after training a sklearn.ensemble.RandomForestClassifier() on my data, I get some leaf nodes where the samples count doesn't match the value array, like this: ...
0
votes
1answer
20 views

Sparse Collaborative Filtering

Does anyone know of any Python code examples for sparse collaborative filtering. Everything I can find revolves around using prebuilt packages (e.g. Mahout, GraphLab), but I'm learning to learn the ...
5
votes
1answer
64 views

Understanding multiple KS tests

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test And this inspired me the following experiment. I ...
2
votes
0answers
141 views

Johansen Test in python

How to get the trace statistics from the Johansen test for cointegration in python. Also, i tried to search it myself on google, found the following website ...
1
vote
1answer
24 views

Recommendation Engine With Physical Distance Cutoff

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based ...